Fixed sidecar collisions, closes #210

typo fix - thanks to @dmd

CHANGELOG.md

@@ -4,6 +4,56 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.33.7](https://github.com/RhetTbull/osxphotos/compare/v0.33.5...v0.33.7)
+
+> 31 August 2020
+
+- typo fix - thanks to @dmd  [`#212`](https://github.com/RhetTbull/osxphotos/pull/212)
+- Normalize unicode for issue #208 [`a36eb41`](https://github.com/RhetTbull/osxphotos/commit/a36eb416b19284477922b6a5f837f4040327138b)
+- Added force_download.py to examples [`b611d34`](https://github.com/RhetTbull/osxphotos/commit/b611d34d19db480af72f57ef55eacd0a32c8d1e8)
+- Added photoshop:SidecarForExtension to XMP, partial fix for #210 [`60d96a8`](https://github.com/RhetTbull/osxphotos/commit/60d96a8f563882fba2365a6ab58c1276725eedaa)
+- Updated README.md [`c9b1518`](https://github.com/RhetTbull/osxphotos/commit/c9b15186a022d91248451279e5f973e3f2dca4b4)
+- Update README.md [`42e8fba`](https://github.com/RhetTbull/osxphotos/commit/42e8fba125a3c6b1bd0d538f2af511aabfbeb478)
+
+#### [v0.33.5](https://github.com/RhetTbull/osxphotos/compare/v0.33.3...v0.33.5)
+
+> 25 August 2020
+
+- Fixed DST handling for from_date/to_date, closes #193 (again) [`#193`](https://github.com/RhetTbull/osxphotos/issues/193)
+- Added raw timestamps to PhotoInfo._info [`0f457a4`](https://github.com/RhetTbull/osxphotos/commit/0f457a4082a4eebc42a5df2160a02ad987b6f96c)
+
+#### [v0.33.3](https://github.com/RhetTbull/osxphotos/compare/v0.33.2...v0.33.3)
+
+> 23 August 2020
+
+- Fixed portrait for Catalina/Big Sur; see issue #203 [`1f717b0`](https://github.com/RhetTbull/osxphotos/commit/1f717b05794c2088c7c15d2aab0c5d24b6309c06)
+
+#### [v0.33.2](https://github.com/RhetTbull/osxphotos/compare/v0.33.0...v0.33.2)
+
+> 23 August 2020
+
+- Closes issue #206, adds --touch-file [`#207`](https://github.com/RhetTbull/osxphotos/pull/207)
+- Touch files - fixes #194 -- thanks to @PabloKohan  [`#205`](https://github.com/RhetTbull/osxphotos/pull/205)
+- Refactor/cleanup _export_photo - thanks to @PabloKohan  [`#204`](https://github.com/RhetTbull/osxphotos/pull/204)
+- Finished --touch-file, closes #206 [`#206`](https://github.com/RhetTbull/osxphotos/issues/206)
+- Merge pull request #205 from PabloKohan/touch_files__fix_194 [`#194`](https://github.com/RhetTbull/osxphotos/issues/194)
+- --touch-file now working with --update [`6c11e3f`](https://github.com/RhetTbull/osxphotos/commit/6c11e3fa5b5b05b98b9fdbb0e59e3a78c7dff980)
+- Refactor/cleanup _export_photo [`eefa1f1`](https://github.com/RhetTbull/osxphotos/commit/eefa1f181f4fd7b027ae69abd2b764afb590c081)
+- Fixed touch tests [`1bf7105`](https://github.com/RhetTbull/osxphotos/commit/1bf7105737fbd756064a2f9ef4d4bbd0b067978c)
+- Working on issue 206 [`ebd878a`](https://github.com/RhetTbull/osxphotos/commit/ebd878a075983ef3df0b1ead1a725e01508721f8)
+- Working on issue #206 [`c9c9202`](https://github.com/RhetTbull/osxphotos/commit/c9c920220545dc27c8cb1379d7bde15987cce72c)
+
+#### [v0.33.0](https://github.com/RhetTbull/osxphotos/compare/v0.32.0...v0.33.0)
+
+> 17 August 2020
+
+- Replaced call to which, closes #171 [`#171`](https://github.com/RhetTbull/osxphotos/issues/171)
+- Added contributors to README.md, closes #200 [`#200`](https://github.com/RhetTbull/osxphotos/issues/200)
+- Added tests for 10.15.6 [`d2deeff`](https://github.com/RhetTbull/osxphotos/commit/d2deefff834e46e1a26adc01b1b025ac839dbc78)
+- Added ImportInfo for Photos 5+ [`98e4170`](https://github.com/RhetTbull/osxphotos/commit/98e417023ec5bd8292b25040d0844f3706645950)
+- Update README.md [`360c8d8`](https://github.com/RhetTbull/osxphotos/commit/360c8d8e1b4760e95a8b71b3a0bf0df4fb5adaf5)
+- Update README.md [`868cda8`](https://github.com/RhetTbull/osxphotos/commit/868cda8482ce6b29dd00e04a209d40550e6b128b)
+
 #### [v0.32.0](https://github.com/RhetTbull/osxphotos/compare/v0.31.2...v0.32.0)
 
 > 9 August 2020

README.md

@@ -53,10 +53,12 @@ OSXPhotos uses setuptools, thus simply run:
 
 	python3 setup.py install
 
-You can also install directly from [pypi](https://pypi.org/):
+You can also install directly from [pypi](https://pypi.org/project/osxphotos/):
 
     pip install osxphotos
 	
+**WARNING** The git repo for this project is very large (> 1GB) because it contains multiple Photos libraries used for testing on different versions of MacOS.  If you just want to use the osxphotos package in your own code, I recommend you install the latest version from [PyPI](https://pypi.org/project/osxphotos/). If you just want to use the command line utility, you can download a pre-built executable of the latest [release](https://github.com/RhetTbull/osxphotos/releases) or you can install via `pip` which also installs the command line app.  If you aren't comfortable with running python on your Mac, start with the pre-built executable.
+
 ## Command Line Usage
 
 This package will install a command line utility called `osxphotos` that allows you to query the Photos database.  Alternatively, you can also run the command line utility like this: `python3 -m osxphotos`
@@ -203,14 +205,14 @@ Options:
                                   both images and movies).
   --only-photos                   Search only for photos/images (default
                                   searches both images and movies).
-  --from-date [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
-                                  Search by start item date, e.g.
-                                  2000-01-12T12:00:00 or 2000-12-31 (ISO 8601
-                                  w/o TZ).
-  --to-date [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
-                                  Search by end item date, e.g.
-                                  2000-01-12T12:00:00 or 2000-12-31 (ISO 8601
-                                  w/o TZ).
+  --from-date DATETIME            Search by start item date, e.g.
+                                  2000-01-12T12:00:00,
+                                  2001-01-12T12:00:00-07:00, or 2000-12-31
+                                  (ISO 8601).
+  --to-date DATETIME              Search by end item date, e.g.
+                                  2000-01-12T12:00:00,
+                                  2001-01-12T12:00:00-07:00, or 2000-12-31
+                                  (ISO 8601).
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -222,6 +224,8 @@ Options:
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
+  --touch-file                    Sets the file's modification time to match
+                                  photo date.
   --overwrite                     Overwrite existing files. Default behavior
                                   is to add (1), (2), etc to filename if file
                                   already exists. Use this with caution as it
@@ -671,7 +675,7 @@ if __name__ == "__main__":
 #### Read a Photos library database
 
 ```python
-osxphotos.PhotosDB() # not recommended, see Note below
+osxphotos.PhotosDB()
 osxphotos.PhotosDB(path)
 osxphotos.PhotosDB(dbfile=path)
 ```
@@ -682,7 +686,7 @@ Pass the path to a Photos library or to a specific database file (e.g. "/Users/s
 
 If an invalid path is passed, PhotosDB will raise `FileNotFoundError` exception.
 
-**Note**: If neither path or dbfile is passed, PhotosDB will use get_last_library_path to open the last opened Photos library.  This usually works but is not 100% reliable.  It can also lead to loading a different library than expected if the user has held down *option* key when opening Photos to switch libraries.  It is therefore recommended you explicitely pass the path to `PhotosDB()`.  
+**Note**: If neither path or dbfile is passed, PhotosDB will use get_last_library_path to open the last opened Photos library.  This usually works but is not 100% reliable.  It can also lead to loading a different library than expected if the user has held down *option* key when opening Photos to switch libraries.  You may therefore want to explicitely pass the path to `PhotosDB()`. 
 
 #### Open the default (last opened) Photos library
 
@@ -1428,8 +1432,8 @@ PhotosDB.import_info returns a list of ImportInfo objects.  Each ImportInfo obje
 #### `uuid`
 Returns the universally unique identifier (uuid) of the import session.  This is how Photos keeps track of individual objects within the database.
 
-#### <a name="albumphotos">`photos`</a>
-Returns a list of [PhotoInfo](#PhotoInfo) objects representing each photo contained in the album sorted in the same order as in Photos. (e.g. if photos were manually sorted in the Photos albums, photos returned by `photos` will be in same order as they appear in the Photos album)
+#### <a name="importphotos">`photos`</a>
+Returns a list of [PhotoInfo](#PhotoInfo) objects representing each photo contained in the import session.
 
 #### `creation_date`
 Returns the creation date as a timezone aware datetime.datetime object of the import session.
@@ -1873,6 +1877,8 @@ Thank-you to the following people who have contributed to improving osxphotos!
 - [Thibault Deutsch](https://github.com/dethi)
 - [grundsch](https://github.com/grundsch)
 - [Ag Primatic](https://github.com/agprimatic)
+- [Daniel M. Drucker](https://github.com/dmd)
+
 
 ## Known Bugs
 
@@ -1888,7 +1894,7 @@ This package works by creating a copy of the sqlite3 database that photos uses t
 
 If apple changes the database format this will likely break.
 
-Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the funcationality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
+Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the functionality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
 
 For additional details about how osxphotos is implemented or if you would like to extend the code, see the [wiki](https://github.com/RhetTbull/osxphotos/wiki).
 

examples/force_download.py

@@ -0,0 +1,42 @@
+""" use osxphotos to force the download of photos from iCloud
+    downloads images to a temporary directory then deletes them
+    resulting in the photo being downloaded to Photos library
+"""
+
+import os
+import sys
+import tempfile
+
+import osxphotos
+
+
+def main():
+    photosdb = osxphotos.PhotosDB()
+    tempdir = tempfile.TemporaryDirectory()
+    photos = photosdb.photos()
+    downloaded = 0
+    missing = [photo for photo in photos if photo.ismissing and not photo.shared]
+
+    if not missing:
+        print(f"Did not find any missing photos to download")
+        sys.exit(0)
+
+    print(f"Downloading {len(missing)} photos")
+    for photo in missing:
+        if photo.ismissing:
+            print(f"Downloading photo {photo.original_filename}")
+            downloaded += 1
+            exported = photo.export(tempdir.name, use_photos_export=True, timeout=300)
+            if photo.hasadjustments:
+                exported.extend(
+                    photo.export(tempdir.name, use_photos_export=True, edited=True, timeout=300)
+                )
+            for filename in exported:
+                print(f"Removing temporary file {filename}")
+                os.unlink(filename)
+    print(f"Downloaded {downloaded} photos")
+    tempdir.cleanup()
+
+
+if __name__ == "__main__":
+    main()

osxphotos/__main__.py

@@ -10,6 +10,7 @@ import pathlib
 import pprint
 import sys
 import time
+import unicodedata
 
 import click
 import yaml
@@ -22,7 +23,7 @@ from pathvalidate import (
 
 import osxphotos
 
-from ._constants import _EXIF_TOOL_URL, _PHOTOS_4_VERSION, _UNKNOWN_PLACE
+from ._constants import _EXIF_TOOL_URL, _PHOTOS_4_VERSION, _UNKNOWN_PLACE, UNICODE_FORMAT
 from ._export_db import ExportDB, ExportDBInMemory
 from ._version import __version__
 from .datetime_formatter import DateTimeFormatter
@@ -40,9 +41,21 @@ OSXPHOTOS_EXPORT_DB = ".osxphotos_export.db"
 
 
 def verbose(*args, **kwargs):
+    """ print output if verbose flag set """
     if VERBOSE:
         click.echo(*args, **kwargs)
 
+def normalize_unicode(value):
+    """ normalize unicode data """
+    if value is not None:
+        if isinstance(value, tuple):
+            return tuple(unicodedata.normalize(UNICODE_FORMAT, v) for v in value)
+        elif isinstance(value, str):
+            return unicodedata.normalize(UNICODE_FORMAT, value)
+        else:
+            return value
+    else:
+        return None
 
 def get_photos_db(*db_options):
     """ Return path to photos db, select first non-None db_options
@@ -1121,6 +1134,11 @@ def query(
     help="Hardlink files instead of copying them.  "
     "Cannot be used with --exiftool which creates copies of the files with embedded EXIF data.",
 )
+@click.option(
+    "--touch-file",
+    is_flag=True,
+    help="Sets the file's modification time to match photo date.",
+)
 @click.option(
     "--overwrite",
     is_flag=True,
@@ -1264,6 +1282,13 @@ def query(
     "to a filesystem that doesn't support Mac OS extended attributes.  Only use this if you get "
     "an error while exporting.",
 )
+@click.option(
+    "--use-photos-export",
+    is_flag=True,
+    default=False,
+    hidden=True,
+    help="Force the use of AppleScript to export even if not missing (see also --download-missing).",
+)
 @DB_ARGUMENT
 @click.argument("dest", nargs=1, type=click.Path(exists=True))
 @click.pass_obj
@@ -1299,6 +1324,7 @@ def export(
     update,
     dry_run,
     export_as_hardlink,
+    touch_file,
     overwrite,
     export_by_date,
     skip_edited,
@@ -1344,6 +1370,7 @@ def export(
     label,
     deleted,
     deleted_only,
+    use_photos_export,
 ):
     """ Export photos from the Photos database.
         Export path DEST is required.
@@ -1514,7 +1541,6 @@ def export(
         deleted_only=deleted_only,
     )
 
-    results_exported = []
     if photos:
         if export_bursts:
             # add the burst_photos to the export set
@@ -1533,10 +1559,12 @@ def export(
         # because the original code used --original-name as an option
         original_name = not current_name
 
+        results_exported = []
         results_new = []
         results_updated = []
         results_skipped = []
         results_exif_updated = []
+        results_touched = []
         if verbose_:
             for p in photos:
                 results = export_photo(
@@ -1564,13 +1592,16 @@ def export(
                     export_db=export_db,
                     fileutil=fileutil,
                     dry_run=dry_run,
+                    touch_file=touch_file,
                     edited_suffix=edited_suffix,
+                    use_photos_export=use_photos_export,
                 )
                 results_exported.extend(results.exported)
                 results_new.extend(results.new)
                 results_updated.extend(results.updated)
                 results_skipped.extend(results.skipped)
                 results_exif_updated.extend(results.exif_updated)
+                results_touched.extend(results.touched)
 
         else:
             # show progress bar
@@ -1601,31 +1632,42 @@ def export(
                         export_db=export_db,
                         fileutil=fileutil,
                         dry_run=dry_run,
+                        touch_file=touch_file,
                         edited_suffix=edited_suffix,
+                        use_photos_export=use_photos_export,
                     )
                     results_exported.extend(results.exported)
                     results_new.extend(results.new)
                     results_updated.extend(results.updated)
                     results_skipped.extend(results.skipped)
                     results_exif_updated.extend(results.exif_updated)
+                    results_touched.extend(results.touched)
         stop_time = time.perf_counter()
         # print summary results
         if update:
             photo_str_new = "photos" if len(results_new) != 1 else "photo"
-            photo_str_updated = "photos" if len(results_new) != 1 else "photo"
+            photo_str_updated = "photos" if len(results_updated) != 1 else "photo"
             photo_str_skipped = "photos" if len(results_skipped) != 1 else "photo"
+            photo_str_touched = "photos" if len(results_touched) != 1 else "photo"
             photo_str_exif_updated = (
                 "photos" if len(results_exif_updated) != 1 else "photo"
             )
-            click.echo(
+            summary = (
                 f"Exported: {len(results_new)} {photo_str_new}, "
-                + f"updated: {len(results_updated)} {photo_str_updated}, "
-                + f"skipped: {len(results_skipped)} {photo_str_skipped}, "
-                + f"updated EXIF data: {len(results_exif_updated)} {photo_str_exif_updated}"
+                f"updated: {len(results_updated)} {photo_str_updated}, "
+                f"skipped: {len(results_skipped)} {photo_str_skipped}, "
+                f"updated EXIF data: {len(results_exif_updated)} {photo_str_exif_updated}"
             )
+            if touch_file:
+                summary += f", touched date: {len(results_touched)} {photo_str_touched}"
+            click.echo(summary)
         else:
             photo_str = "photos" if len(results_exported) != 1 else "photo"
-            click.echo(f"Exported: {len(results_exported)} {photo_str}")
+            photo_str_touched = "photos" if len(results_touched) != 1 else "photo"
+            summary = f"Exported: {len(results_exported)} {photo_str}"
+            if touch_file:
+                summary += f", touched date: {len(results_touched)} {photo_str_touched}"
+            click.echo(summary)
         click.echo(f"Elapsed time: {(stop_time-start_time):.3f} seconds")
     else:
         click.echo("Did not find any photos to export")
@@ -1834,6 +1876,15 @@ def _query(
             to_date=to_date,
         )
 
+    person = normalize_unicode(person)
+    keyword = normalize_unicode(keyword)
+    album = normalize_unicode(album)
+    folder = normalize_unicode(folder)
+    title = normalize_unicode(title)
+    description = normalize_unicode(description)
+    place = normalize_unicode(place)
+    label = normalize_unicode(label)
+    
     if album:
         photos = get_photos_by_attribute(photos, "albums", album, ignore_case)
 
@@ -2073,7 +2124,9 @@ def export_photo(
     export_db=None,
     fileutil=FileUtil,
     dry_run=None,
+    touch_file=None,
     edited_suffix="_edited",
+    use_photos_export=False,
 ):
     """ Helper function for export that does the actual export
 
@@ -2101,6 +2154,8 @@ def export_photo(
         export_db: export database instance compatible with ExportDB_ABC
         fileutil: file util class compatible with FileUtilABC
         dry_run: boolean; if True, doesn't actually export or update any files
+        touch_file: boolean; sets file's modification time to match photo date
+        use_photos_export: boolean; if True forces the use of AppleScript to export even if photo not missing
 
     Returns:
         list of path(s) of exported photo or None if photo was missing
@@ -2115,25 +2170,26 @@ def export_photo(
         if photo.ismissing:
             space = " " if not verbose_ else ""
             verbose(f"{space}Skipping missing photo {photo.filename}")
-            return ExportResults([], [], [], [], [])
+            return ExportResults([], [], [], [], [], [])
         elif not os.path.exists(photo.path):
             space = " " if not verbose_ else ""
             verbose(
                 f"{space}WARNING: file {photo.path} is missing but ismissing=False, "
                 f"skipping {photo.filename}"
             )
-            return ExportResults([], [], [], [], [])
+            return ExportResults([], [], [], [], [], [])
     elif photo.ismissing and not photo.iscloudasset or not photo.incloud:
         verbose(
             f"Skipping missing {photo.filename}: not iCloud asset or missing from cloud"
         )
-        return ExportResults([], [], [], [], [])
+        return ExportResults([], [], [], [], [], [])
 
     results_exported = []
     results_new = []
     results_updated = []
     results_skipped = []
     results_exif_updated = []
+    results_touched = []
 
     filenames = get_filenames_from_template(photo, filename_template, original_name)
     for filename in filenames:
@@ -2152,8 +2208,10 @@ def export_photo(
 
         # if download_missing and the photo is missing or path doesn't exist,
         # try to download with Photos
-        use_photos_export = download_missing and (
-            photo.ismissing or not os.path.exists(photo.path)
+        use_photos_export = (
+            download_missing and (photo.ismissing or not os.path.exists(photo.path))
+            if not use_photos_export
+            else True
         )
 
         # export the photo to each path in dest_paths
@@ -2178,6 +2236,7 @@ def export_photo(
                 export_db=export_db,
                 fileutil=fileutil,
                 dry_run=dry_run,
+                touch_file=touch_file,
             )
 
             results_exported.extend(export_results.exported)
@@ -2185,6 +2244,7 @@ def export_photo(
             results_updated.extend(export_results.updated)
             results_skipped.extend(export_results.skipped)
             results_exif_updated.extend(export_results.exif_updated)
+            results_touched.extend(export_results.touched)
 
             if verbose_:
                 for exported in export_results.exported:
@@ -2195,13 +2255,14 @@ def export_photo(
                     verbose(f"Exported updated file {updated}")
                 for skipped in export_results.skipped:
                     verbose(f"Skipped up to date file {skipped}")
+                for touched in export_results.touched:
+                    verbose(f"Touched date on file {touched}")
 
             # if export-edited, also export the edited version
             # verify the photo has adjustments and valid path to avoid raising an exception
             if export_edited and photo.hasadjustments:
                 # if download_missing and the photo is missing or path doesn't exist,
                 # try to download with Photos
-                use_photos_export = download_missing and photo.path_edited is None
                 if not download_missing and photo.path_edited is None:
                     verbose(f"Skipping missing edited photo for {filename}")
                 else:
@@ -2234,6 +2295,7 @@ def export_photo(
                         export_db=export_db,
                         fileutil=fileutil,
                         dry_run=dry_run,
+                        touch_file=touch_file,
                     )
 
                     results_exported.extend(export_results_edited.exported)
@@ -2241,6 +2303,7 @@ def export_photo(
                     results_updated.extend(export_results_edited.updated)
                     results_skipped.extend(export_results_edited.skipped)
                     results_exif_updated.extend(export_results_edited.exif_updated)
+                    results_touched.extend(export_results_edited.touched)
 
                     if verbose_:
                         for exported in export_results_edited.exported:
@@ -2251,6 +2314,8 @@ def export_photo(
                             verbose(f"Exported updated file {updated}")
                         for skipped in export_results_edited.skipped:
                             verbose(f"Skipped up to date file {skipped}")
+                        for touched in export_results_edited.touched:
+                            verbose(f"Touched date on file {touched}")
 
     return ExportResults(
         results_exported,
@@ -2258,6 +2323,7 @@ def export_photo(
         results_updated,
         results_skipped,
         results_exif_updated,
+        results_touched,
     )
 
 

osxphotos/_constants.py

@@ -9,6 +9,9 @@ from datetime import datetime
 # Apple Epoch is Jan 1, 2001
 TIME_DELTA = (datetime(2001, 1, 1, 0, 0) - datetime(1970, 1, 1, 0, 0)).total_seconds()
 
+# Unicode format to use for comparing strings
+UNICODE_FORMAT = "NFC"
+
 # which Photos library database versions have been tested
 # Photos 2.0 (10.12.6) == 2622
 # Photos 3.0 (10.13.6) == 3301
@@ -43,6 +46,7 @@ _DB_TABLE_NAMES = {
         "ALBUM_JOIN": "Z_26ASSETS.Z_34ASSETS",
         "ALBUM_SORT_ORDER": "Z_26ASSETS.Z_FOK_34ASSETS",
         "IMPORT_FOK": "ZGENERICASSET.Z_FOK_IMPORTSESSION",
+        "DEPTH_STATE": "ZGENERICASSET.ZDEPTHSTATES",
     },
     6: {
         "ASSET": "ZASSET",
@@ -50,6 +54,7 @@ _DB_TABLE_NAMES = {
         "ALBUM_JOIN": "Z_26ASSETS.Z_3ASSETS",
         "ALBUM_SORT_ORDER": "Z_26ASSETS.Z_FOK_3ASSETS",
         "IMPORT_FOK": "null",
+        "DEPTH_STATE": "ZASSET.ZDEPTHTYPE",
     },
 }
 

osxphotos/_export_db.py

@@ -189,7 +189,12 @@ class ExportDB(ExportDB_ABC):
                 (filename,),
             )
             results = c.fetchone()
-            stats = results[0:3] if results else None
+            if results:
+                stats = results[0:3]
+                mtime = int(stats[2]) if stats[2] is not None else None
+                stats = (stats[0], stats[1], mtime)
+            else:
+                stats = (None, None, None)
         except Error as e:
             logging.warning(e)
             stats = (None, None, None)
@@ -232,7 +237,12 @@ class ExportDB(ExportDB_ABC):
                 (filename,),
             )
             results = c.fetchone()
-            stats = results[0:3] if results else None
+            if results:
+                stats = results[0:3]
+                mtime = int(stats[2]) if stats[2] is not None else None
+                stats = (stats[0], stats[1], mtime)
+            else:
+                stats = (None, None, None)
         except Error as e:
             logging.warning(e)
             stats = (None, None, None)

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.33.0"
+__version__ = "0.33.8"

osxphotos/albuminfo.py

@@ -57,7 +57,9 @@ class AlbumInfoBaseClass:
         self._creation_date_timestamp = self._db._dbalbum_details[uuid]["creation_date"]
         self._start_date_timestamp = self._db._dbalbum_details[uuid]["start_date"]
         self._end_date_timestamp = self._db._dbalbum_details[uuid]["end_date"]
-        self._local_tz = get_local_tz()
+        self._local_tz = get_local_tz(
+            datetime.fromtimestamp(self._creation_date_timestamp + TIME_DELTA)
+        )
 
     @property
     def uuid(self):

osxphotos/datetime_utils.py

@@ -2,14 +2,23 @@
 
 import datetime
 
-def get_local_tz():
-    """ return local timezone as datetime.timezone tzinfo """
-    local_tz = (
-        datetime.datetime.now(datetime.timezone(datetime.timedelta(0)))
-        .astimezone()
-        .tzinfo
-    )
-    return local_tz
+
+def get_local_tz(dt):
+    """ return local timezone as datetime.timezone tzinfo for dt
+    
+    Args:
+        dt: datetime.datetime
+    
+    Returns:
+        local timezone for dt as datetime.timezone
+
+    Raises:
+        ValueError if dt is not timezone naive
+    """
+    if not datetime_has_tz(dt):
+        return dt.astimezone().tzinfo
+    else:
+        raise ValueError("dt must be naive datetime.datetime object")
 
 
 def datetime_remove_tz(dt):
@@ -50,4 +59,4 @@ def datetime_naive_to_local(dt):
             f"{dt} has tzinfo {dt.tzinfo} and offset {dt.tizinfo.utcoffset(dt)}"
         )
 
-    return dt.replace(tzinfo=get_local_tz())
+    return dt.replace(tzinfo=get_local_tz(dt))

osxphotos/fileutil.py

@@ -29,7 +29,17 @@ class FileUtilABC(ABC):
 
     @classmethod
     @abstractmethod
-    def cmp_sig(cls, file1, file2):
+    def utime(cls, path, times):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def cmp(cls, file1, file2, mtime1=None):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def cmp_file_sig(cls, file1, file2):
         pass
 
     @classmethod
@@ -104,11 +114,37 @@ class FileUtilMacOS(FileUtilABC):
             os.unlink(filepath)
 
     @classmethod
-    def cmp_sig(cls, f1, s2):
+    def utime(cls, path, times):
+        """ Set the access and modified time of path. """
+        os.utime(path, times)
+
+    @classmethod
+    def cmp(cls, f1, f2, mtime1=None):
+        """Does shallow compare (file signatures) of f1 to file f2.
+        Arguments:
+        f1 --  File name
+        f2 -- File name
+        mtime1 -- optional, pass alternate file modification timestamp for f1; will be converted to int
+
+        Return value:
+        True if the file signatures as returned by stat are the same, False otherwise. 
+        Does not do a byte-by-byte comparison.
+        """
+
+        s1 = cls._sig(os.stat(f1))
+        if mtime1 is not None:
+            s1 = (s1[0], s1[1], int(mtime1))
+        s2 = cls._sig(os.stat(f2))
+        if s1[0] != stat.S_IFREG or s2[0] != stat.S_IFREG:
+            return False
+        return s1 == s2
+
+    @classmethod
+    def cmp_file_sig(cls, f1, s2):
         """Compare file f1 to signature s2.
         Arguments:
         f1 --  File name
-        s2 -- stats as returned by sig
+        s2 -- stats as returned by _sig
 
         Return value:
         True if the files are the same, False otherwise.
@@ -130,7 +166,12 @@ class FileUtilMacOS(FileUtilABC):
 
     @staticmethod
     def _sig(st):
-        return (stat.S_IFMT(st.st_mode), st.st_size, st.st_mtime)
+        """ return tuple of (mode, size, mtime) of file based on os.stat
+            Args:
+                st: os.stat signature
+        """
+        # use int(st.st_mtime) because ditto does not copy fractional portion of mtime
+        return (stat.S_IFMT(st.st_mode), st.st_size, int(st.st_mtime))
 
 
 class FileUtil(FileUtilMacOS):
@@ -141,8 +182,8 @@ class FileUtil(FileUtilMacOS):
 
 class FileUtilNoOp(FileUtil):
     """ No-Op implementation of FileUtil for testing / dry-run mode
-        all methods with exception of cmp_sig and file_cmp are no-op
-        cmp_sig functions as FileUtil.cmp_sig does
+        all methods with exception of cmp, cmp_file_sig and file_cmp are no-op
+        cmp and cmp_file_sig functions as FileUtil methods do
         file_cmp returns mock data
     """
 
@@ -172,6 +213,10 @@ class FileUtilNoOp(FileUtil):
     def unlink(cls, dest):
         cls.verbose(f"unlink: {dest}")
 
+    @classmethod
+    def utime(cls, path, times):
+        cls.verbose(f"utime: {path}, {times}")
+
     @classmethod
     def file_sig(cls, file1):
         cls.verbose(f"file_sig: {file1}")

osxphotos/photoinfo/_photoinfo_export.py

@@ -11,7 +11,6 @@
 
 # TODO: should this be its own PhotoExporter class?
 
-import filecmp
 import glob
 import json
 import logging
@@ -37,7 +36,8 @@ from ..fileutil import FileUtil
 from ..utils import dd_to_dms_str, findfiles
 
 ExportResults = namedtuple(
-    "ExportResults", ["exported", "new", "updated", "skipped", "exif_updated"]
+    "ExportResults",
+    ["exported", "new", "updated", "skipped", "exif_updated", "touched"],
 )
 
 
@@ -305,6 +305,7 @@ def export2(
     export_db=None,
     fileutil=FileUtil,
     dry_run=False,
+    touch_file=False,
 ):
     """ export photo, like export but with update and dry_run options
         dest: must be valid destination path or exception raised 
@@ -347,6 +348,7 @@ def export2(
                 for getting/setting data related to exported files to compare update state
         fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
         dry_run: (boolean, default=False); set to True to run in "dry run" mode
+        touch_file: (boolean, default=False); if True, sets file's modification time upon photo date
 
         Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
                     where each field is a list of file paths
@@ -375,6 +377,9 @@ def export2(
     # list of all files skipped because they do not need to be updated (for use with update=True)
     update_skipped_files = []
 
+    # list of all files with utime touched (touch_file = True)
+    touched_files = []
+
     # check edited and raise exception trying to export edited version of
     # photo that hasn't been edited
     if edited and not self.hasadjustments:
@@ -482,7 +487,7 @@ def export2(
         if update and dest.exists():
             # destination exists, check to see if destination is the right UUID
             dest_uuid = export_db.get_uuid_for_file(dest)
-            if dest_uuid is None and filecmp.cmp(src, dest):
+            if dest_uuid is None and fileutil.cmp(src, dest):
                 # might be exporting into a pre-ExportDB folder or the DB got deleted
                 logging.debug(
                     f"Found matching file with blank uuid: {self.uuid}, {dest}"
@@ -514,7 +519,7 @@ def export2(
                         dest = pathlib.Path(file_)
                         found_match = True
                         break
-                    elif dest_uuid is None and filecmp.cmp(src, file_):
+                    elif dest_uuid is None and fileutil.cmp(src, file_):
                         # files match, update the UUID
                         logging.debug(
                             f"Found matching file with blank uuid: {self.uuid}, {file_}"
@@ -558,12 +563,14 @@ def export2(
             no_xattr,
             export_as_hardlink,
             exiftool,
+            touch_file,
             fileutil,
         )
         exported_files = results.exported
         update_new_files = results.new
         update_updated_files = results.updated
         update_skipped_files = results.skipped
+        touched_files = results.touched
 
         # copy live photo associated .mov if requested
         if live_photo and self.live_photo:
@@ -583,12 +590,14 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
+                    touch_file,
                     fileutil,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
+                touched_files.extend(results.touched)
             else:
                 logging.debug(f"Skipping missing live movie for {filename}")
 
@@ -608,17 +617,19 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
+                    touch_file,
                     fileutil,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
+                touched_files.extend(results.touched)
             else:
                 logging.debug(f"Skipping missing RAW photo for {filename}")
     else:
         # use_photo_export
-        exported = None
+        exported = []
         # export live_photo .mov file?
         live_photo = True if live_photo and self.live_photo else False
         if edited:
@@ -628,7 +639,7 @@ def export2(
                 filestem = dest.stem
             else:
                 # didn't get passed a filename, add _edited
-                filestem = f"{dest.stem}_edited"
+                filestem = f"{dest.stem}{edited_identifier}"
                 dest = dest.parent / f"{filestem}.jpeg"
 
             exported = _export_photo_uuid_applescript(
@@ -657,8 +668,16 @@ def export2(
                 dry_run=dry_run,
             )
 
-        if exported is not None:
+        if exported:
+            if touch_file:
+                for exported_file in exported:
+                    touched_files.append(exported_file)
+                    ts = int(self.date.timestamp())
+                    fileutil.utime(exported_file, (ts, ts))
             exported_files.extend(exported)
+            if update:
+                update_new_files.extend(exported)
+
         else:
             logging.warning(
                 f"Error exporting photo {self.uuid} to {dest} with use_photos_export"
@@ -669,7 +688,7 @@ def export2(
 
     if sidecar_json:
         logging.debug("writing exiftool_json_sidecar")
-        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.json")
+        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.json")
         sidecar_str = self._exiftool_json_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
             use_persons_as_keywords=use_persons_as_keywords,
@@ -685,12 +704,13 @@ def export2(
 
     if sidecar_xmp:
         logging.debug("writing xmp_sidecar")
-        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.xmp")
+        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.xmp")
         sidecar_str = self._xmp_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
             use_persons_as_keywords=use_persons_as_keywords,
             keyword_template=keyword_template,
             description_template=description_template,
+            extension=dest.suffix[1:] if dest.suffix else None,
         )
         if not dry_run:
             try:
@@ -760,6 +780,7 @@ def export2(
                     keyword_template=keyword_template,
                     description_template=description_template,
                 )
+
             export_db.set_exifdata_for_file(
                 exported_file,
                 self._exiftool_json_sidecar(
@@ -774,13 +795,23 @@ def export2(
             )
             exif_files_updated.append(exported_file)
 
-    return ExportResults(
+    if touch_file:
+        for exif_file in exif_files_updated:
+            touched_files.append(exported_file)
+            ts = int(self.date.timestamp())
+            fileutil.utime(exported_file, (ts, ts))
+
+    touched_files = list(set(touched_files))
+
+    results = ExportResults(
         exported_files,
         update_new_files,
         update_updated_files,
         update_skipped_files,
         exif_files_updated,
+        touched_files,
     )
+    return results
 
 
 def _export_photo(
@@ -793,11 +824,12 @@ def _export_photo(
     no_xattr,
     export_as_hardlink,
     exiftool,
+    touch_file,
     fileutil=FileUtil,
 ):
     """ Helper function for export()
         Does the actual copy or hardlink taking the appropriate 
-        action depending on update, overwrite
+        action depending on update, overwrite, export_as_hardlink
         Assumes destination is the right destination (e.g. UUID matches)
         sets UUID and JSON info foo exported file using set_uuid_for_file, set_inf_for_uuido
     
@@ -810,6 +842,7 @@ def _export_photo(
         no_xattr: don't copy extended attributes
         export_as_hardlink: bool
         exiftool: bool
+        touch_file: bool
         fileutil: FileUtil class that conforms to fileutil.FileUtilABC
 
     Returns:
@@ -820,143 +853,99 @@ def _export_photo(
     update_updated_files = []
     update_new_files = []
     update_skipped_files = []
+    touched_files = []
 
     dest_str = str(dest)
     dest_exists = dest.exists()
     if export_as_hardlink:
-        # use hardlink instead of copy
-        if not update:
-            # not update, do the the hardlink
-            if overwrite and dest.exists():
-                # need to remove the destination first
-                # dest.unlink()
-                fileutil.unlink(dest)
-            logging.debug(f"Not update: export_as_hardlink linking file {src} {dest}")
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-        elif dest_exists and dest.samefile(src):
-            # update, hardlink and it already points to the right file, do nothing
-            logging.debug(
-                f"Update: skipping samefile with export_as_hardlink {src} {dest}"
-            )
-            update_skipped_files.append(dest_str)
-        elif dest_exists:
-            # update, not the same file (e.g. user may not have used export_as_hardlink last time it was run
-            logging.debug(
-                f"Update: removing existing file prior to export_as_hardlink {src} {dest}"
-            )
-            # dest.unlink()
-            fileutil.unlink(dest)
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            update_updated_files.append(dest_str)
-            exported_files.append(dest_str)
-        else:
-            # update, hardlink, destination doesn't exist (new file)
-            logging.debug(
-                f"Update: exporting new file with export_as_hardlink {src} {dest}"
-            )
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-            update_new_files.append(dest_str)
+        op_desc = "export_as_hardlink"
     else:
-        if not update:
-            # not update, do the the copy
-            if overwrite and dest.exists():
-                # need to remove the destination first
-                # dest.unlink()
-                fileutil.unlink(dest)
-            logging.debug(f"Not update: copying file {src} {dest}")
-            fileutil.copy(src, dest_str, norsrc=no_xattr)
-            exported_files.append(dest_str)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-        # elif dest_exists and not exiftool and cmp_file(dest_str, export_db.get_stat_orig_for_file(dest_str)):
-        elif (
-            dest_exists
-            and not exiftool
-            and filecmp.cmp(src, dest)
-            and not dest.samefile(src)
-        ):
-            # destination exists but is identical
-            logging.debug(f"Update: skipping identifical original files {src} {dest}")
-            # call set_stat because code can reach this spot if no export DB but exporting a RAW or live photo
-            # potentially re-writes the data in the database but ensures database is complete
-            export_db.set_stat_orig_for_file(dest_str, fileutil.file_sig(dest_str))
-            update_skipped_files.append(dest_str)
-        elif (
-            dest_exists
-            and exiftool
-            and fileutil.cmp_sig(dest_str, export_db.get_stat_exif_for_file(dest_str))
-            and not dest.samefile(src)
-        ):
-            # destination exists but is identical
-            logging.debug(f"Update: skipping identifical exiftool files {src} {dest}")
-            update_skipped_files.append(dest_str)
-        elif dest_exists:
-            # destination exists but is different or is a hardlink
-            logging.debug(f"Update: removing existing file prior to copy {src} {dest}")
-            stat_src = os.stat(src)
-            stat_dest = os.stat(dest)
-            # dest.unlink()
-            fileutil.unlink(dest)
-            fileutil.copy(src, dest_str, norsrc=no_xattr)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-            update_updated_files.append(dest_str)
-        else:
-            # destination doesn't exist, copy the file
-            logging.debug(f"Update: copying new file {src} {dest}")
-            fileutil.copy(src, dest_str, norsrc=no_xattr)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
+        op_desc = "export_by_copying"
+
+    if not update:
+        # not update, export the file
+        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
+        exported_files.append(dest_str)
+        if touch_file:
+            sig = fileutil.file_sig(src)
+            sig = (sig[0], sig[1], int(self.date.timestamp()))
+            if not fileutil.cmp_file_sig(src, sig):
+                touched_files.append(dest_str)
+    else:  # updating
+        if not dest_exists:
+            # update, destination doesn't exist (new file)
+            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
             update_new_files.append(dest_str)
+            if touch_file:
+                touched_files.append(dest_str)
+        else:
+            # update, destination exists, but we might not need to replace it...
+            if exiftool:
+                sig_exif = export_db.get_stat_exif_for_file(dest_str)
+                cmp_orig = fileutil.cmp_file_sig(dest_str, sig_exif)
+                sig_exif = (sig_exif[0], sig_exif[1], int(self.date.timestamp()))
+                cmp_touch = fileutil.cmp_file_sig(dest_str, sig_exif)
+            else:
+                cmp_orig = fileutil.cmp(src, dest)
+                cmp_touch = fileutil.cmp(src, dest, mtime1=int(self.date.timestamp()))
+
+            sig_cmp = cmp_touch if touch_file else cmp_orig
+
+            if (export_as_hardlink and dest.samefile(src)) or (
+                not export_as_hardlink and not dest.samefile(src) and sig_cmp
+            ):
+                # destination exists and signatures match, skip it
+                update_skipped_files.append(dest_str)
+            else:
+                # destination exists but signature is different
+                if touch_file and cmp_orig and not cmp_touch:
+                    # destination exists, signature matches original but does not match expected touch time
+                    # skip exporting but update touch time
+                    update_skipped_files.append(dest_str)
+                    touched_files.append(dest_str)
+                elif not touch_file and cmp_touch and not cmp_orig:
+                    # destination exists, signature matches expected touch but not original
+                    # user likely exported with touch_file and is now exporting without touch_file
+                    # don't update the file because it's same but leave touch time
+                    update_skipped_files.append(dest_str)
+                else:
+                    # destination exists but is different
+                    update_updated_files.append(dest_str)
+                    if touch_file:
+                        touched_files.append(dest_str)
+
+    if not update_skipped_files:
+        if dest_exists and (update or overwrite):
+            # need to remove the destination first
+            logging.debug(
+                f"Update: removing existing file prior to {op_desc} {src} {dest}"
+            )
+            fileutil.unlink(dest)
+        if export_as_hardlink:
+            fileutil.hardlink(src, dest)
+        else:
+            fileutil.copy(src, dest_str, norsrc=no_xattr)
+
+        export_db.set_data(
+            dest_str,
+            self.uuid,
+            fileutil.file_sig(dest_str),
+            (None, None, None),
+            self.json(),
+            None,
+        )
+
+    if touched_files:
+        ts = int(self.date.timestamp())
+        fileutil.utime(dest, (ts, ts))
 
     return ExportResults(
-        exported_files, update_new_files, update_updated_files, update_skipped_files, []
+        exported_files + update_new_files + update_updated_files,
+        update_new_files,
+        update_updated_files,
+        update_skipped_files,
+        [],
+        touched_files,
     )
 
 
@@ -1132,6 +1121,7 @@ def _xmp_sidecar(
     use_persons_as_keywords=False,
     keyword_template=None,
     description_template=None,
+    extension=None
 ):
     """ returns string for XMP sidecar 
         use_albums_as_keywords: treat album names as keywords
@@ -1139,10 +1129,12 @@ def _xmp_sidecar(
         keyword_template: (list of strings); list of template strings to render as keywords 
         description_template: string; optional template string that will be rendered for use as photo description """
 
-    # TODO: add additional fields to XMP file?
-
     xmp_template = Template(filename=os.path.join(_TEMPLATE_DIR, _XMP_TEMPLATE_NAME))
 
+    if extension is None:
+        extension = pathlib.Path(self.original_filename)
+        extension = extension.suffix[1:] if extension.suffix else None
+        
     if description_template is not None:
         description = self.render_template(
             description_template, expand_inplace=True, inplace_sep=", "
@@ -1211,6 +1203,7 @@ def _xmp_sidecar(
         keywords=keyword_list,
         persons=person_list,
         subjects=subject_list,
+        extension=extension,
     )
 
     # remove extra lines that mako inserts from template

osxphotos/photosdb/_photosdb_process_faceinfo.py

@@ -4,7 +4,7 @@
 import logging
 
 from .._constants import _DB_TABLE_NAMES, _PHOTOS_4_VERSION
-from ..utils import _open_sql_file
+from ..utils import _open_sql_file, normalize_unicode
 from .photosdb_utils import get_db_version
 
 
@@ -121,7 +121,7 @@ def _process_faceinfo_4(photosdb):
         face["asset_uuid"] = asset_uuid
         face["uuid"] = row[2]
         face["person"] = person_id
-        face["fullname"] = row[3]
+        face["fullname"] = normalize_unicode(row[3])
         face["sourcewidth"] = row[7]
         face["sourceheight"] = row[8]
         face["centerx"] = row[9]
@@ -282,7 +282,7 @@ def _process_faceinfo_5(photosdb):
         face["asset_uuid"] = asset_uuid
         face["uuid"] = row[2]
         face["person"] = person_pk
-        face["fullname"] = row[4]
+        face["fullname"] = normalize_unicode(row[4])
         face["agetype"] = row[5]
         face["baldtype"] = row[6]
         face["eyemakeuptype"] = row[7]

osxphotos/photosdb/_photosdb_process_searchinfo.py

@@ -10,7 +10,7 @@ import uuid as uuidlib
 from pprint import pformat
 
 from .._constants import _PHOTOS_4_VERSION, SEARCH_CATEGORY_LABEL
-from ..utils import _db_is_locked, _debug, _open_sql_file
+from ..utils import _db_is_locked, _debug, _open_sql_file, normalize_unicode
 
 """
     This module should be imported in the class defintion of PhotosDB in photosdb.py
@@ -112,8 +112,8 @@ def _process_searchinfo(self):
         record["groupid"] = row[3]
         record["category"] = row[4]
         record["owning_groupid"] = row[5]
-        record["content_string"] = row[6].replace("\x00", "")
-        record["normalized_string"] = row[7].replace("\x00", "")
+        record["content_string"] = normalize_unicode(row[6].replace("\x00", ""))
+        record["normalized_string"] = normalize_unicode(row[7].replace("\x00", ""))
         record["lookup_identifier"] = row[8]
 
         try:
@@ -147,9 +147,10 @@ def _process_searchinfo(self):
             "_db_searchinfo_labels_normalized: \n"
             + pformat(self._db_searchinfo_labels_normalized)
         )
-    
+
     conn.close()
 
+
 @property
 def labels(self):
     """ return list of all search info labels found in the library """

osxphotos/photosdb/photosdb.py

@@ -44,6 +44,7 @@ from ..utils import (
     _get_os_version,
     _open_sql_file,
     get_last_library_path,
+    normalize_unicode,
 )
 from .photosdb_utils import get_db_model_version, get_db_version
 
@@ -713,7 +714,7 @@ class PhotosDB:
         for album in c:
             self._dbalbum_details[album[0]] = {
                 "_uuid": album[0],
-                "title": album[1],
+                "title": normalize_unicode(album[1]),
                 "cloudlibrarystate": album[2],
                 "cloudidentifier": album[3],
                 "intrash": False if album[4] == 0 else True,
@@ -760,7 +761,7 @@ class PhotosDB:
             self._dbfolder_details[uuid] = {
                 "_uuid": row[0],
                 "modelId": row[1],
-                "name": row[2],
+                "name": normalize_unicode(row[2]),
                 "isMagic": row[3],
                 "intrash": row[4],
                 "folderType": row[5],
@@ -932,6 +933,7 @@ class PhotosDB:
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
+            self._dbphotos[uuid]["lastmodifieddate_timestamp"] = row[4]
             try:
                 self._dbphotos[uuid]["lastmodifieddate"] = datetime.fromtimestamp(
                     row[4] + TIME_DELTA
@@ -942,6 +944,7 @@ class PhotosDB:
                 self._dbphotos[uuid]["lastmodifieddate"] = None
 
             self._dbphotos[uuid]["imageTimeZoneOffsetSeconds"] = row[9]
+            self._dbphotos[uuid]["imageDate_timestamp"] = row[5]
 
             try:
                 imagedate = datetime.fromtimestamp(row[5] + TIME_DELTA)
@@ -961,7 +964,7 @@ class PhotosDB:
             self._dbphotos[uuid]["volumeId"] = row[10]
             self._dbphotos[uuid]["imagePath"] = row[11]
             self._dbphotos[uuid]["extendedDescription"] = row[12]
-            self._dbphotos[uuid]["name"] = row[13]
+            self._dbphotos[uuid]["name"] = normalize_unicode(row[13])
             self._dbphotos[uuid]["isMissing"] = row[14]
             self._dbphotos[uuid]["originalFilename"] = row[15]
             self._dbphotos[uuid]["favorite"] = row[16]
@@ -1448,6 +1451,7 @@ class PhotosDB:
         album_join = _DB_TABLE_NAMES[photos_ver]["ALBUM_JOIN"]
         album_sort = _DB_TABLE_NAMES[photos_ver]["ALBUM_SORT_ORDER"]
         import_fok = _DB_TABLE_NAMES[photos_ver]["IMPORT_FOK"]
+        depth_state = _DB_TABLE_NAMES[photos_ver]["DEPTH_STATE"]
 
         # Look for all combinations of persons and pictures
         if _debug():
@@ -1605,7 +1609,7 @@ class PhotosDB:
         for album in c:
             self._dbalbum_details[album[0]] = {
                 "_uuid": album[0],
-                "title": album[1],
+                "title": normalize_unicode(album[1]),
                 "cloudlocalstate": album[2],
                 "cloudownerfirstname": album[3],
                 "cloudownderlastname": album[4],
@@ -1680,12 +1684,13 @@ class PhotosDB:
                 JOIN ZKEYWORD ON ZKEYWORD.Z_PK = {keyword_join} """
         )
         for keyword in c:
+            keyword_title = normalize_unicode(keyword[0])
             if not keyword[1] in self._dbkeywords_uuid:
                 self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword[0] in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword[0]] = []
+            if not keyword_title in self._dbkeywords_keyword:
+                self._dbkeywords_keyword[keyword_title] = []
             self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword[0]].append(keyword[1])
+            self._dbkeywords_keyword[keyword_title].append(keyword[1])
 
         if _debug():
             logging.debug(f"Finished walking through keywords")
@@ -1739,7 +1744,8 @@ class PhotosDB:
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALHEIGHT, 
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALWIDTH, 
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
-                ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE
+                ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE,
+                {depth_state}
                 FROM {asset_table} 
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = {asset_table}.Z_PK 
                 ORDER BY {asset_table}.ZUUID  """
@@ -1782,6 +1788,7 @@ class PhotosDB:
         # 33   ZADDITIONALASSETATTRIBUTES.ZORIGINALWIDTH,
         # 34   ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
         # 35   ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE
+        # 36   ZGENERICASSET.ZDEPTHSTATES / ZASSET.ZDEPTHTYPE
 
         for row in c:
             uuid = row[0]
@@ -1790,11 +1797,12 @@ class PhotosDB:
             info["modelID"] = None
             info["masterUuid"] = None
             info["masterFingerprint"] = row[1]
-            info["name"] = row[2]
+            info["name"] = normalize_unicode(row[2])
 
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
+            info["lastmodifieddate_timestamp"] = row[4]
             try:
                 info["lastmodifieddate"] = datetime.fromtimestamp(row[4] + TIME_DELTA)
             except ValueError:
@@ -1803,6 +1811,7 @@ class PhotosDB:
                 info["lastmodifieddate"] = None
 
             info["imageTimeZoneOffsetSeconds"] = row[6]
+            info["imageDate_timestamp"] = row[5]
 
             try:
                 imagedate = datetime.fromtimestamp(row[5] + TIME_DELTA)
@@ -1901,10 +1910,10 @@ class PhotosDB:
             # 3 = HDR photo
             # 4 = non-HDR version of the photo
             # 6 = panorama
-            # 8 = portrait
+            # > 6 = portrait (sometimes, see ZDEPTHSTATE/ZDEPTHTYPE)
             info["customRenderedValue"] = row[22]
             info["hdr"] = True if row[22] == 3 else False
-            info["portrait"] = True if row[22] == 8 else False
+            info["portrait"] = True if row[36] != 0 else False
 
             # Set panorama from either KindSubType or RenderedValue
             info["panorama"] = True if row[21] == 1 or row[22] == 6 else False
@@ -2020,7 +2029,7 @@ class PhotosDB:
         for row in c:
             uuid = row[0]
             if uuid in self._dbphotos:
-                self._dbphotos[uuid]["extendedDescription"] = row[1]
+                self._dbphotos[uuid]["extendedDescription"] = normalize_unicode(row[1])
             else:
                 if _debug():
                     logging.debug(

osxphotos/placeinfo.py

@@ -11,6 +11,9 @@ from collections import namedtuple  # pylint: disable=syntax-error
 import yaml
 from bpylist import archiver
 
+from ._constants import UNICODE_FORMAT
+from .utils import normalize_unicode
+
 # postal address information, returned by PlaceInfo.address
 PostalAddress = namedtuple(
     "PostalAddress",
@@ -76,12 +79,12 @@ class PLRevGeoLocationInfo:
         geoServiceProvider,
         postalAddress,
     ):
-        self.addressString = addressString
+        self.addressString = normalize_unicode(addressString)
         self.countryCode = countryCode
         self.mapItem = mapItem
         self.isHome = isHome
-        self.compoundNames = compoundNames
-        self.compoundSecondaryNames = compoundSecondaryNames
+        self.compoundNames = normalize_unicode(compoundNames)
+        self.compoundSecondaryNames = normalize_unicode(compoundSecondaryNames)
         self.version = version
         self.geoServiceProvider = geoServiceProvider
         self.postalAddress = postalAddress
@@ -183,7 +186,7 @@ class PLRevGeoMapItemAdditionalPlaceInfo:
 
     def __init__(self, area, name, placeType, dominantOrderType):
         self.area = area
-        self.name = name
+        self.name = normalize_unicode(name)
         self.placeType = placeType
         self.dominantOrderType = dominantOrderType
 
@@ -232,13 +235,13 @@ class CNPostalAddress:
         _subLocality,
     ):
         self._ISOCountryCode = _ISOCountryCode
-        self._city = _city
-        self._country = _country
-        self._postalCode = _postalCode
-        self._state = _state
-        self._street = _street
-        self._subAdministrativeArea = _subAdministrativeArea
-        self._subLocality = _subLocality
+        self._city = normalize_unicode(_city)
+        self._country = normalize_unicode(_country)
+        self._postalCode = normalize_unicode(_postalCode)
+        self._state = normalize_unicode(_state)
+        self._street = normalize_unicode(_street)
+        self._subAdministrativeArea = normalize_unicode(_subAdministrativeArea)
+        self._subLocality = normalize_unicode(_subLocality)
 
     def __eq__(self, other):
         return all(
@@ -414,9 +417,9 @@ class PlaceInfo4(PlaceInfo):
             #   2: type
             #   3: area
             try:
-                places_dict[p[2]].append((p[1], p[3]))
+                places_dict[p[2]].append((normalize_unicode(p[1]), p[3]))
             except KeyError:
-                places_dict[p[2]] = [(p[1], p[3])]
+                places_dict[p[2]] = [(normalize_unicode(p[1]), p[3])]
 
         # build list to populate PlaceNames tuple
         # initialize with empty lists for each field in PlaceNames

osxphotos/templates/xmp_sidecar.mako

@@ -1,5 +1,13 @@
 <!-- Created with osxphotos https://github.com/RhetTbull/osxphotos -->
 
+<%def name="photoshop_sidecar_for_extension(extension)">
+    % if extension is None:
+        <photoshop:SidecarForExtension></photoshop:SidecarForExtension>
+    % else:
+        <photoshop:SidecarForExtension>${extension}</photoshop:SidecarForExtension>
+    % endif
+</%def>
+
 <%def name="dc_description(desc)">
     % if desc is None:
         <dc:description></dc:description>
@@ -86,6 +94,7 @@
         <rdf:Description rdf:about="" 
             xmlns:dc="http://purl.org/dc/elements/1.1/" 
             xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
+            ${photoshop_sidecar_for_extension(extension)}
             ${dc_description(description)}
             ${dc_title(photo.title)}
             ${dc_subject(subjects)}

osxphotos/utils.py

@@ -10,6 +10,7 @@ import sqlite3
 import subprocess
 import sys
 import tempfile
+import unicodedata
 import urllib.parse
 from plistlib import load as plistload
 
@@ -18,6 +19,7 @@ import CoreServices
 import objc
 from Foundation import *
 
+from ._constants import UNICODE_FORMAT
 from .fileutil import FileUtil
 
 _DEBUG = False
@@ -352,3 +354,13 @@ def _db_is_locked(dbname):
 #     attr = xattr.xattr(filepath)
 #     uuid_bytes = bytes(uuid, 'utf-8')
 #     attr.set(OSXPHOTOS_XATTR_UUID, uuid_bytes)
+
+
+def normalize_unicode(value):
+    """ normalize unicode data """
+    if value is not None:
+        if not isinstance(value, str):
+            raise ValueError("value must be str")
+        return unicodedata.normalize(UNICODE_FORMAT, value)
+    else:
+        return None

tests/README.md

@@ -17,6 +17,9 @@ Some of the export tests rely on photos in my local library and will look for `O
 
 One test for locale does not run on GitHub's automated workflow and will look for `OSXPHOTOS_TEST_LOCALE=1` to determine if it should be run.  If you want to run this test, set the environment variable.  
 
+## Test Photo Libraries
+**Important**: The test code uses several test photo libraries created on various version of MacOS.  If you need to inspect one of these or modify one for a test, make a copy of the library (for example, copy it to your ~/Pictures folder) then open the copy in Photos.  Once done, copy the revised library back to the tests/ folder.  If you do not do this, the Photos background process photoanalysisd will forever try to process the library resulting in updates to the database which will cause git to see changes to the file you didn't intend.  I'm not aware of any way to disassociate photoanalysisd from the library once you've opened it in Photos.
+
 ## Attribution ##
 These tests utilize a test Photos library. The test library is populated with photos from [flickr](https://www.flickr.com) and from my own photo library.  All images used are licensed under Creative Commons 2.0 Attribution [license](https://creativecommons.org/licenses/by/2.0/).  
 

tests/Test-10.15.6.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>1847</integer>
+	<integer>1942</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-Cloud-10.16.0.photoslibrary/database/DataModelVersion.plist

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>LibrarySchemaVersion</key>
+	<integer>5001</integer>
+	<key>MetaSchemaVersion</key>
+	<integer>3</integer>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/database/Photos.sqlite.lock

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>hostname</key>
+	<string>Rhets-MacBook-Pro.local</string>
+	<key>hostuuid</key>
+	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
+	<key>pid</key>
+	<integer>73945</integer>
+	<key>processname</key>
+	<string>photolibraryd</string>
+	<key>uid</key>
+	<integer>502</integer>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/database/search/searchMetadata.plist

@@ -0,0 +1,203 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BlacklistedMeaningsByMeaning</key>
+	<dict>
+		<key>Brunch</key>
+		<array>
+			<string>Dining</string>
+		</array>
+		<key>Brunches</key>
+		<array>
+			<string>Dining</string>
+		</array>
+		<key>Lunch</key>
+		<array>
+			<string>Dining</string>
+		</array>
+		<key>Lunches</key>
+		<array>
+			<string>Dining</string>
+		</array>
+	</dict>
+	<key>SceneWhitelist</key>
+	<array>
+		<string>Graduation</string>
+		<string>Aquarium</string>
+		<string>Food</string>
+		<string>Ice Skating</string>
+		<string>Mountain</string>
+		<string>Cliff</string>
+		<string>Basketball</string>
+		<string>Tennis</string>
+		<string>Jewelry</string>
+		<string>Cheese</string>
+		<string>Softball</string>
+		<string>Football</string>
+		<string>Circus</string>
+		<string>Jet Ski</string>
+		<string>Playground</string>
+		<string>Carousel</string>
+		<string>Paint Ball</string>
+		<string>Windsurfing</string>
+		<string>Sailboat</string>
+		<string>Sunbathing</string>
+		<string>Dam</string>
+		<string>Fireplace</string>
+		<string>Flower</string>
+		<string>Scuba</string>
+		<string>Hiking</string>
+		<string>Cetacean</string>
+		<string>Pier</string>
+		<string>Bowling</string>
+		<string>Snowboarding</string>
+		<string>Zoo</string>
+		<string>Snowmobile</string>
+		<string>Theater</string>
+		<string>Boat</string>
+		<string>Casino</string>
+		<string>Car</string>
+		<string>Diving</string>
+		<string>Cycling</string>
+		<string>Musical Instrument</string>
+		<string>Board Game</string>
+		<string>Castle</string>
+		<string>Sunset Sunrise</string>
+		<string>Martial Arts</string>
+		<string>Motocross</string>
+		<string>Submarine</string>
+		<string>Cat</string>
+		<string>Snow</string>
+		<string>Kiteboarding</string>
+		<string>Squash</string>
+		<string>Geyser</string>
+		<string>Music</string>
+		<string>Archery</string>
+		<string>Desert</string>
+		<string>Blackjack</string>
+		<string>Fireworks</string>
+		<string>Sportscar</string>
+		<string>Feline</string>
+		<string>Soccer</string>
+		<string>Museum</string>
+		<string>Baby</string>
+		<string>Fencing</string>
+		<string>Railroad</string>
+		<string>Nascar</string>
+		<string>Sky Surfing</string>
+		<string>Bird</string>
+		<string>Games</string>
+		<string>Baseball</string>
+		<string>Dressage</string>
+		<string>Snorkeling</string>
+		<string>Pyramid</string>
+		<string>Kite</string>
+		<string>Rowboat</string>
+		<string>Golf</string>
+		<string>Watersports</string>
+		<string>Lightning</string>
+		<string>Canyon</string>
+		<string>Auditorium</string>
+		<string>Night Sky</string>
+		<string>Karaoke</string>
+		<string>Skiing</string>
+		<string>Parade</string>
+		<string>Forest</string>
+		<string>Hot Air Balloon</string>
+		<string>Dragon Parade</string>
+		<string>Easter Egg</string>
+		<string>Monument</string>
+		<string>Jungle</string>
+		<string>Thanksgiving</string>
+		<string>Jockey Horse</string>
+		<string>Stadium</string>
+		<string>Airplane</string>
+		<string>Ballet</string>
+		<string>Yoga</string>
+		<string>Coral Reef</string>
+		<string>Skating</string>
+		<string>Wrestling</string>
+		<string>Bicycle</string>
+		<string>Tattoo</string>
+		<string>Amusement Park</string>
+		<string>Canoe</string>
+		<string>Cheerleading</string>
+		<string>Ping Pong</string>
+		<string>Fishing</string>
+		<string>Magic</string>
+		<string>Reptile</string>
+		<string>Winter Sport</string>
+		<string>Waterfall</string>
+		<string>Train</string>
+		<string>Bonsai</string>
+		<string>Surfing</string>
+		<string>Dog</string>
+		<string>Cake</string>
+		<string>Sledding</string>
+		<string>Sandcastle</string>
+		<string>Glacier</string>
+		<string>Lighthouse</string>
+		<string>Equestrian</string>
+		<string>Rafting</string>
+		<string>Shore</string>
+		<string>Hockey</string>
+		<string>Santa Claus</string>
+		<string>Formula One Car</string>
+		<string>Sport</string>
+		<string>Vehicle</string>
+		<string>Boxing</string>
+		<string>Rollerskating</string>
+		<string>Underwater</string>
+		<string>Orchestra</string>
+		<string>Carnival</string>
+		<string>Rocket</string>
+		<string>Skateboarding</string>
+		<string>Helicopter</string>
+		<string>Performance</string>
+		<string>Oktoberfest</string>
+		<string>Water Polo</string>
+		<string>Skate Park</string>
+		<string>Animal</string>
+		<string>Nightclub</string>
+		<string>String Instrument</string>
+		<string>Dinosaur</string>
+		<string>Gymnastics</string>
+		<string>Cricket</string>
+		<string>Volcano</string>
+		<string>Lake</string>
+		<string>Aurora</string>
+		<string>Dancing</string>
+		<string>Concert</string>
+		<string>Rock Climbing</string>
+		<string>Hang Glider</string>
+		<string>Rodeo</string>
+		<string>Fish</string>
+		<string>Art</string>
+		<string>Motorcycle</string>
+		<string>Volleyball</string>
+		<string>Wake Boarding</string>
+		<string>Badminton</string>
+		<string>Motor Sport</string>
+		<string>Sumo</string>
+		<string>Parasailing</string>
+		<string>Skydiving</string>
+		<string>Kickboxing</string>
+		<string>Pinata</string>
+		<string>Foosball</string>
+		<string>Go Kart</string>
+		<string>Poker</string>
+		<string>Kayak</string>
+		<string>Swimming</string>
+		<string>Atv</string>
+		<string>Beach</string>
+		<string>Dartboard</string>
+		<string>Athletics</string>
+		<string>Camping</string>
+		<string>Tornado</string>
+		<string>Billiards</string>
+		<string>Rugby</string>
+		<string>Airshow</string>
+	</array>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/database/search/searchProgress.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>insertAlbum</key>
+	<array/>
+	<key>insertAsset</key>
+	<array/>
+	<key>insertHighlight</key>
+	<array/>
+	<key>insertMemory</key>
+	<array/>
+	<key>insertMoment</key>
+	<array/>
+	<key>removeAlbum</key>
+	<array/>
+	<key>removeAsset</key>
+	<array/>
+	<key>removeHighlight</key>
+	<array/>
+	<key>removeMemory</key>
+	<array/>
+	<key>removeMoment</key>
+	<array/>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/database/search/searchSystemInfo.plist

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>embeddingVersion</key>
+	<string>1</string>
+	<key>localeIdentifier</key>
+	<string>en_US</string>
+	<key>sceneTaxonomySHA</key>
+	<string>4abe27e61f90a30eb22be0b6b052bfb7ed2aab82a24f956d43a5aba57965f50e</string>
+	<key>searchIndexVersion</key>
+	<string>15000</string>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>CollapsedSidebarSectionIdentifiers</key>
+	<array/>
+	<key>ExpandedSidebarItemIdentifiers</key>
+	<array>
+		<string>PXMediaTypesVirtualCollection</string>
+	</array>
+	<key>IPXWorkspaceControllerZoomLevelsKey</key>
+	<dict>
+		<key>kZoomLevelIdentifierPhotosGrid</key>
+		<integer>1</integer>
+	</dict>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/private/com.apple.mediaanalysisd/caches/vision/PersonPromoter

@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>NumberOfFacesProcessedOnLastRun</key>
+	<integer>3</integer>
+	<key>ProcessedInQuiescentState</key>
+	<true/>
+	<key>Version</key>
+	<integer>4</integer>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/private/com.apple.mediaanalysisd/caches/vision/clustererState.plist

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>PVClustererBringUpState</key>
+	<integer>40</integer>
+</dict>
+</plist>

tests/Test-Cloud-10.16.0.photoslibrary/private/com.apple.mediaanalysisd/caches/vision/faceWorkerState.plist

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>PersonBuilderLastMinimumFaceGroupSizeForCreatingMergeCandidates</key>
+	<integer>15</integer>
+	<key>PersonBuilderMergeCandidatesEnabled</key>
+	<true/>
+</dict>
+</plist>