--------------------------------------------------------------------------- Understanding Syncing: How Updates, Imports, and Deletions Work in this App --------------------------------------------------------------------------- Different people work on different computers on different copies of the same database. Syncing connects the separate backend databases by combining their data. The import database is merged into the local database using a structured syncing process that works with the following key tables: - Baptism Book Table - Church Records Table - All Contribution Tables The App distinguishes three types of sync: Import, update and delete sync. When you choose FULL SYNC, the App will run the import and update sync, leaving out the delete sync until you engage it separately. But you can also run each sync independently of the other. The App will identify for you all new records (unique in the import database) and all overlapping records that were changed on the other computer. By browsing through these records, you can control the syncing process in an exact way. --> For the import sync, all new records need to be imported together in order to keep the relationships between the records alive (parents cards, secondary cards, etc). --> For the update sync, you can view local and import records side by side, with the differences pointed out exactly. If you browse through the records, you can opt to update there and then, record by record. Or you can update all at once in bulk. -------------------- The basis of syncing -------------------- Records are identified by the App as being identical across different databases through the so-called GUID (Globally Unique ID) or the Creation Time Stamp. When a record is created, it is given its own unique GUID that cannot be changed, and also a creation time stamp. If two Mary Bandas have the same GUID across two versions of the same database, then we are dealing with the same person. If GUIDs differ, then we are dealing with two persons of the same name, Mary Banda. If Mary Banda is called in another database Maria Annastazia Banda but has the same GUID, she is still recognised as the same person. Meaning: her name was changed/corrected on the other computer. But if the same Mary Banda has independently been entered into two databases (hence with different GUIDs), the duplicate record should not be imported. The App can realise this through the baptism number as the ultimate ground of syncing. ------------------------------------------------------------------ 1. Import Sync: Adding New Records from the import database to the local database ------------------------------------------------------------------ When a record from the import database has a unique GUID (meaning it does not exist in the local database), the system considers it a potential new record. However, there are two possibilities: - The record is truly new and needs to be imported into the local database. - The record was deleted in the local database and should not be re-imported. To handle this, the App keeps a memory of all deleted records. - If the record exists in this memory, it will not be imported again. - If it is not in the memory, it will be imported as a new record. Note: You can manually clear the memory of deleted records (RESET LOCAL DELETE MEMORY). If you do, all records in the import database that do not exist locally will be treated as new and will be imported - including the records you deleted on purpose. IMPORTANT - WHEN IT IS SAFE TO CLEAR THE DELETE MEMORY: only when no copy you will still sync with holds a record you deleted. A version made before your deletion still contains the deleted record, and once the memory is empty that record comes back as new. In practice, clear the memory only at one of these moments: - At the end of a complete round of syncing: every participating computer has either run its own delete sync, or has had its datafile replaced by the newly synced copy (or has downloaded the new master), and no older version is still waiting to be synced - on a memory stick or handed in on the server. - At a natural break, for example the end of the year, once all copies have been brought together in this way and a fresh round starts. Never clear it after syncing with only one of several versions, or while other members may still hand in versions made before the deletions. If you are unsure, keep the memory: keeping it longer does no harm, it only takes a little space. --------------------------------------------- 2. Update Sync: Keeping the Most Recent Edits --------------------------------------------- The sync operation compares records in both databases based on their GUID (or the Creation Time Stamp, if you wish). If a record exists in both databases, the sync process checks if any changes were made on the other computer to this record. If changes are detected in any field, then it looks at the Edit Time Stamp to determine which version is more up-to-date. - If the record in the import database was edited more recently, its data will replace the version in the local database. - If the local database has the most recent edit time stamp, it will keep its version and ignore the import. Note that there is a user option by means of which you can override the edit stamp. In that case, all changes of the import database will be copied into the local database, even if more recent changes were done on the local database. By default, this option is switched off. On the partial sync panel, you find an additional dedicated command for updating contribution tables. Again the rules about the edit time stamp applies. ---------------------------------------- 3. Delete Sync: Removing Deleted Records ---------------------------------------- The sync also handles deletions: - If a record that exists in the local database is missing from the import database, the sync process will check the import's deleted records memory. - If the record is listed in the deleted memory, the delete sync action will delete the record from the local database as well. Every database keeps such a memory of its own. When you delete a person, a baptism entry or a financial entry here, the App writes down that it was deleted, and two things follow from that. A later sync with a copy that still holds the record will NOT bring it back: the deleted memory outranks the import. And the copy you sync with can be offered the same deletion, so that a deletion travels through the team instead of being quietly undone by the next person who syncs. Note: To avoid false deletions, nothing is ever deleted without being shown to you first. All concerned records are listed; you delete them one by one, or all of them together after one confirmation that names how many there are. FINANCIAL ENTRIES (the ledgers) The individual ledger (LedgerI) and the parish ledger (LedgerP) used to travel one way only: new entries were imported, but an entry deleted on another computer stayed here for ever, and an entry you had deleted here came back with the next sync. They now follow the same rules as the records above. On the SYNC form, the two rows DELETE LEDGER-I and DELETE LEDGER-P count the entries that the import database deleted and that you still have. SHOW & DELETE lists them as they stand in YOUR ledger - date, accounts, amount, year, person - because by then the entry no longer exists in the import database to be looked at. The X beside a row deletes that one entry here as well; DELETE ALL LISTED deletes every listed entry after one confirmation. Whatever you delete this way goes into your own deleted memory, so it carries on to the next computer you sync with. What counts as a deletion, and what does not: - A deletion: the X on any ledger form, deleting a person (their financial entries go with them), the REPLACE mode of the contribution grid (the entries it replaces really do disappear), and the entries dropped when two duplicate records are merged. - NOT a deletion, and never offered to anybody as one: archiving a year (the entries move into the archive - they are not gone) and converting an account between the individual and the parish ledger (the entry moves from one ledger to the other and keeps its identity). A computer that has not archived the same year must never be told to delete those entries. An import database made by an older Parish Record Keeper carries no ledger deleted memory at all. That is not an error: it simply knows of no deletions, and the two rows stay empty. In the same way, deletions you make here will not reach somebody still running an older version. ---------------- Advanced options ---------------- These options apply mainly for bulk processing syncs, meaning all the dark green and light green buttons. (1) Make log file This should always be checked. The logfile will inform you exactly which records were imported and which records were updated or deleted. The logfile will also point out any failure or inconsistencies in the data structure that may interfere with the sync process. (2) Import new records of spouses or children as cardholders upon failure to map them securely with existing local records The sync process matches the cards of both databases. Problems arise when the same card has changed in both databases, but in different ways. For example, in the import database, Mary Banda, a new record (not present in the local database), is married to Paul Phiri, who is present in both databases. But in the local database, Paul Phiri is married to a different wife. If the option is ticked, Mary Banda will be imported on a new card, without a husband. If not ticked, she will not be imported. In either case, the log file will report if she has been imported, and where to. (3) Allow imports that create non-unique baptism numbers in the Baptism Book Table The import table has a new record, and yet the local database contains already another record with the same baptism number. While this can occur in the church records system, in which we register also people who were baptised in different parishes or who are unbaptised, it should not occur in the baptism book table. By nature, all baptism numbers of the same Parish should be different. If checked (not recommended), the file will be nevertheless imported into the baptism book table. Note: The App also contains checks for double baptism numbers. CHURCH FILES DUPLICATES on the control panel has a DOUBLE BAPT NOs button that retrieves every church file whose baptism number occurs more than once in this parish and opens exactly those records for correction; BAPTISM BOOK DUPLICATES does the same work on the baptism register itself. This makes the task of working on them and addressing the issue very easy. (4) Ignore edit time stamp on update (Not recommended). Changes were done in the local database and in the import database on the same record. But the changes differ! Which one to take? If you check this option, the sync process will always choose the changes of the import database over the local database. Otherwise, it gives priority to the most recent edit time stamp. (5) Sync-field options The default sync field is the creation time stamp - "Ground sync in Creation Time (recommended)" on the SYNC form. Usually, you should go with the default option. The reason: when a baptism entry is linked to a church record, the two are given the SAME GUID, so a GUID no longer tells them apart - the creation time stamp always remains unique. The App sets it automatically when a record is created. Unlike GUIDs, the application allows you to manipulate creation time stamps as well as edit time stamps in bulk, which gives you a chance to include or exclude sections of large imports from the syncing process. COMPARE ON THE SERVER matches records by the creation time stamp too. The alternative is the Globally Unique ID (GUID) - "Ground sync in GUID". It is generated automatically when a record is created and cannot be changed, unless you override it manually in a table (which defeats its purpose of existence) - but because linked records share it (see above), it is the second choice. Choose it when the creation time stamps of a datafile can no longer be trusted, for example after they were changed in bulk. Finally, but only for the baptism book table, you can also choose the Baptism Number as the sync-field. When you change the sync-field, the App will first check if there are duplicates in that sync-field, which qualifies it. In that case, an error message will be displayed. ------------ How to sync? ------------ A NOTE FIRST: what follows is the memory-stick route, and it is still exactly right for a parish that works without the internet. A parish with a workspace on parishrecordkeeper.com has a shorter way round the same circle - one published master on the server that everybody downloads, and versions handed in to be merged rather than carried about - and it is described in "What a workspace gives the parish" (Documentation\WorkspaceBenefits.txt) and "The Server Actions window" (Documentation\ServerActions.txt). The syncing machinery below is what does the merging in either case. There, the SYNC form opens by itself with the downloaded file already loaded (DOWNLOAD MASTER AND SYNC, DOWNLOAD SELECTED AND SYNC), so steps 2 and 3 below fall away. There is also a way without the SYNC form: COMPARE ON THE SERVER, which follows the same rules as the sync below - the same matching of records, the same edit time stamp and the same "ignore edit time" option, the same delete memory, the same way of finding a new person's card - and delivers the differences as change notes: corrections, new families with their links to other cards, persons moved to other cards, new baptism entries and deletions. Which of the two ways to use is the subject of the next section. -------------------------------------------------- Which way to sync: on the server, or with the SYNC form -------------------------------------------------- A parish with a workspace has two ways of merging a member's version. Both follow the rules of this document; they differ in how much you see and in what happens afterwards. ONLINE SYNC - COMPARE ON THE SERVER, then apply the change notes. RECOMMENDED WHEN YOU TRUST THE SOURCE - a member whose work you know, a version you expect to be right. Tick what to take and apply everything in one go: corrections, new families (with their cards, card of origin and secondary cards), persons moved to other cards, new baptism entries. Only notes that carry a warning, and every deletion, are still shown one by one. The same window then keeps a dated copy of your datafile before anything is written, and afterwards publishes the result as the new MASTER - with the online search database if you wish. The member's version is marked as merged, and every team member gets the new master when they next start the App (automatically, if their datafile is set to take new masters automatically). One pass from the member's version to everybody's datafile. Documentation\ServerActions.txt describes each step. OFFLINE SYNC - the SYNC form (DOWNLOAD SELECTED AND SYNC, or a file on a memory stick). RECOMMENDED FOR A DETAILED, SIDE-BY-SIDE COMPARISON - a version you want to check record by record, a large or unfamiliar import, a datafile that was set up separately, or when you want to decide on each new person and each changed field with both versions in front of you. It shows local and import records side by side with the differences pointed out, and offers every option described below. It also works without the internet. Publishing the result is offered when the SYNC form closes. EVERYDAY SYNC - SYNC ALL ON SERVER, for a team that works on the same parish all the time. Nobody sends a whole datafile: each computer sends only what it changed since its last round, as change notes, and the administrator takes those in and publishes the master. Choose it when the changes are corrections, new financial entries and deletions. Choose one of the two above when the work was a REORGANISATION - new, merged or re-scoped accounts, new, merged or moved areas, a repair that rewrites GUIDs or time stamps, or many new families - because change notes cannot carry that, and the button will tell you so rather than try. Documentation\ServerActions.txt describes each step, what happens when two people changed the same thing, and when the button refuses. Both keep every record's identity (GUID and creation time stamp), so a version merged one way can later be compared or synced the other way without duplicates. A new card always gets the next card number of YOUR datafile; the member's card number is never used - cards are recognised by their own identity (card GUID) instead. The process is easy: 1) Make a backup of your present local database. 2) Copy the different versions from the other computers to a memory stick. 3) Upon opening the sync command through the control panel, you are asked to point at the import database file to your memory stick. 4) It now loads automatically copies of all required tables into your own database. 5) The App then makes automatic calculations, detecting and displaying the number of new unique records in the import database that are missing in your own local database. It also points out the number of records that exist in different versions in both databases. And the records that were deleted in the import database but that are still present in your own database. 6) You can now view all these records, browse through them, and decide which ones to import or update and which ones to leave out of the sync session. Or you can opt for a full sync: This will import all new data and update all changed data in accordance with your preferences. 7) At the end, make also a delete sync. 8) Repeat steps 3 to 7 for each version you brought on the memory stick. Do NOT reset the memory of deleted records between them: a version you have not yet synced may still hold records you deleted, and with an empty memory they would be imported again as new. 9) Now copy your backend back to the memory stick and replace all backend copies on the other computers with the newly synced copy. Only now - when every version has been synced and every computer carries the new copy - may you reset the memory of deleted records to start afresh for the next round (or leave it until a natural break such as the end of the year; keeping it does no harm). RESET DELETE MEMORY beside the church records clears the memory of the deleted FINANCIAL ENTRIES with it: a person's entries are deleted together with the person, so the two halves of one deletion are reset together. The baptism book keeps its own button. Emptying a deleted memory means that every record in it, if it is still present in an import file, will be offered to you again as a new record - harmless when every copy has already dropped the record, and exactly the problem in the middle of a round. If a computer did not take the new copy, it still holds the deleted records: keep the memory until it has. 10) When leaving the sync panel, all imported tables will automatically be deleted to avoid unnecessary bloating of the database. 11) Compact the new backend. Then run the table checks from the REPAIR options to make sure that no inconsistencies were imported. ------------ Good to know ------------ The syncing process only affects the local database (master file). The import database itself is not touched or affected in any way. After syncing different import databases into the local master file, you can substitute all the different backends with a copy of the local master file. ------------------------------------------- The Backend ID: exports, syncs and workspaces ------------------------------------------- Besides the GUID of each record, every datafile carries one GUID of its own: the Backend ID (Backend GUID), shown in the ABOUT window. It is how the server on parishrecordkeeper.com recognises a datafile as belonging to a workspace (see "IF THE DATAFILE IS NOT REGISTERED" in Documentation\ServerActions.txt). ALL EXPORTS AND IMPORTS KEEP THE SAME BACKEND ID. Every export on the Export screen - the full database (.PRK), the full text (_PRK.TXT), Excel, the mini database and mini Excel, JSON, SQLite and MySQL - carries the Backend ID of the datafile it came from. When one of them is turned back into a datafile on the Import screen (from the text file, from Excel, from the mini database or from mini Excel), the new datafile takes over that same Backend ID instead of receiving a new one. The same holds for a text file downloaded from the website. An exported or re-imported copy therefore still belongs to the same workspace as the original: it can be used for that workspace - uploaded, handed in, downloaded again and synced - without registering anything a second time. (The one file that is not a copy of the datafile, EXPORT MINI AS TEXT FOR WEB, only feeds the online search and is not meant to be imported.) A DIFFERENT BACKEND ID DOES NOT STOP A SYNC. Syncing works record by record, on the records' own GUIDs and creation time stamps, not on the Backend ID. So a datafile with a different Backend ID - one that was set up separately, on another computer or from a new, empty datafile - can still be synced INTO a datafile whose Backend ID the server recognises as belonging to a workspace. Its records then become part of the workspace datafile and travel with it from then on. (The only difference: contribution accounts from a datafile with a different Backend ID are matched by their names, and the App says so, so it is worth looking over the account structure afterwards.) What does NOT happen is the other way round: syncing a workspace datafile into a datafile with a different Backend ID does not make that datafile part of the workspace. The sync only moves records; the Backend ID of the local datafile stays what it was. Always sync INTO the datafile the server knows.