How To Purge a Dataset

GENERAL INSTRUCTIONS FOR PURGING A DATASET

GOAL:

1. To decrease the size of the dataset so that copying/backup of dataset takes less time & disk space.

2. To decrease the time it takes to run reports by decreasing the number of records that must be searched through to obtain reports

3. Improve processing performance of active transactions.

CHECKLIST:

• Do you have enough space to run the purge? There must be 1.5 times the largest file size of free space to purge the dataset.
  This is a Pervasive (PSQL) Database requirement. PSQL purge will always make a new file, copy over the records that do not meet the purge criteria record by record, then delete the old file. Thus the reason for the needed harddrive space.
  Microsoft SQL (MSQL) database does not require the extra space. MSQL purge will delete the rows directly from the table if they meet the purge criteria. However, if the database has all the audit features turned on, there is a risk that the harddrive could fill up just from audit logs. The hard drive could run out of space as the log files fill up from each deletion of a purged record. See your MSQL administrator to determine your database settings.

• Do you have enough time to run the purge? By Package look at the total file size. Rough calculations are 4mg/minute to purge.(Example: ACP package data files total 98-mg file size divided by 4-mg/min = 25-minutes to purge the ACP package.)

• Have you made a backup of the dataset for off-site storage? Once the dataset is purged, the only way to see the data purged will be through the backup copy. https://culturatech.atlassian.net/wiki/spaces/ACD/pages/3693248526

• Have you made a backup copy of the dataset via Copy Dataset function?
  AGRIS Utilities > Copy Dataset > Follow Process
  Rename Dataset to identify the date ranges it covers via Maintenance > Dataset Maintenance.
  Set security on dataset to View_Only for packages Users normally have access to via Customize > System Security.(Note that Payroll & Ledger may have more limited access)

• If you are using Data Transfer module (DTR) to move data from this dataset to another dataset, has the DTR been completed? Once the dataset is purged, the records that would normally be transferred are deleted. So DTR process must be run prior to purge.

• Has the SJI been run & posted to the ledger for the date range being purged?

• Has Patronage accumulation been run for the date range being purged?

• Has the cost adjustment process in Inventory package been run?

• Have the packages been balanced to the ledger for the periods being purged? Do not purge a period that has not been balanced to the ledger.

RULES TO PURGE:

1. Packages can be purged in any order except for ACP-ACR-INV. ACP & ACR must be purged before INV.

2. Transactions in Bank need to be reconciled (marked with an X ) or the ACP Disbursements or Grain Settlements will not purge, even if they meet purge criteria.

3. Everyone must be out of the dataset during purge.

4. Print purge reports to the spool.

SPECIAL NOTES:

1. The dataset can be purged over a time span.For example, purge ACR & ACP on Monday night.Purge INV on Tuesday night.Purge GRN on Wednesday night.Purge Bank & SJI on Thursday night.

2. If any record in any file is eligible for purging, but has an Original Date, Activity Date, Ledger Date, or Transaction Date associated with it; that is later than the purge date, it will not be purged.

3. Packages to NOT purge: Name Address, Payroll, Patronage, Ledger

HOW TO PURGE:

1. ACP > Purge Data Files >Take Note of the Warning Message Screens. 
   Note that if integrated with the bank, transactions must be reconciled before disbursements will purge.
   Verify that you have followed those steps. >Answer YES to each data type being purged >Enter the purge date > Leave all range parameters blank >Select Report Spool Printer > Purge Starts NOTE: Voucher that meet all purge criteria but are still marked Estimate status will not purge.

2. ACR > Purge Data Files >Take Note of the Warning Message Screens.  Verify that you have followed those steps. >Answer YES to each data type being purged (Historical Balances may be left for Credit history Tracking by answering No) >Enter the purge date > Leave all range parameters blank >Select Report Spool Printer > Purge Starts

3. INV > Purge Data Files >Take Note of the Warning Message Screens.  Verify that you have followed those steps. > Answer YES to each data type being purged >Enter the purge date > Leave all range parameters blank >Select Report Spool Printer > Purge Starts

4. AGY > No purge option.

5. BIN > No purge option.

6. BNK > Purge Data Files >Take Note of the Warning Message Screens.  Verify that you have followed those steps. >Answer YES to each data type being purged > Enter the purge date > Leave all range parameters blank >Select Report Spool Printer > Purge Starts NOTE: Transactions must be reconciled to purge. Reconciliation date is not looked at for purge comparison. So a transaction reconciled with today's date will still purge based on ledger date.

7. CFT > Purge Data Files > Take Note of the Warning Message screens.
   Answer YES to each communication Log to be purged > Enter the purge date > Leave all range parameters blank > select report spool printer > purge starts.

8. DTR > No purge option

9. GRN > Purge Data Files >Take Note of the Warning Message Screens. 
   Note that if integrated with bank, transactions must be reconciled for settlement to purge.
   Verify that you have followed those steps. >Answer YES to each data type being purged >Enter the purge date > Leave all range parameters blank >Select Report Spool Printer > Purge Starts

10. LDG > Purged during Fiscal Close function

11. NAM > Suggestion is to not purge. If you do want to purge name ID must be made inactive or temporary in order to purge

12. ONE > No purge option

13. PAT > Do not Purge

14. PAY > Do not purge.

15. SJI > Purge Data Files >Take Note of the Warning Message Screens.  Verify that you have followed those steps. >Answer YES to each data type being purged except 3) Ledger Detail files (that purges when LDG fiscal close is run) > Leave all range parameters blank >Select Report Spool Printer > Purge Starts NOTE: SJI Periods must be marked closed in order to purge & no open SJI entries can exist for those periods that are purging.

16. SYS > Maintenance > File Maintenance > Purge Data Files >Take Note of the Warning Message Screens.  Verify that you have followed those steps. >Answer YES to each data type being purged >Answer NO to Purge Document Tracking (You can purge it but it is purging based on when the record was added, not if the document it refers to is gone from the dataset. We have another way to purge this.) >Enter Purge Date > Enter to begin purge > Option to select printer for purge reports will appear AFTER the Deleted Records File is purged > Purge of the rest of the files starts.

17. Report Spool:
    Reports > Report Spool > Filter >Key in date ranges of Blank to Purge Date >All reports that are older than purge date will display > Select All > Delete
    NOTE: If you want to keep certain reports, like SJI creation reports, then they can be excluded from the purge/delete by using the range selection criteria to work around that package or report type.

18. Form Spool:
    Reports > Form Spool > Filter >Key in date ranges of Blank to Purge Date >All forms that are older than purge date will display > Select All > Delete
    NOTE: Purging these documents will stop them from displaying in ACR > Balance Inquiry > Go > Forms Spool