Fix for issue #234

Convert to jpeg

CHANGELOG.md

@@ -4,6 +4,144 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.35.1](https://github.com/RhetTbull/osxphotos/compare/v0.35.0...v0.35.1)
+
+> 12 October 2020
+
+- Fix for issue #230 [`dcbf8f2`](https://github.com/RhetTbull/osxphotos/commit/dcbf8f25f61e21bcf1040046aa9d6ddba4ac9735)
+
+#### [v0.35.0](https://github.com/RhetTbull/osxphotos/compare/v0.34.5...v0.35.0)
+
+> 12 October 2020
+
+- Convert to jpeg [`#233`](https://github.com/RhetTbull/osxphotos/pull/233)
+- Updated tests, closes #231 [`#231`](https://github.com/RhetTbull/osxphotos/issues/231)
+- Updated tests [`b0171ba`](https://github.com/RhetTbull/osxphotos/commit/b0171ba6f5b73e1ff71e16d27852f8df7f208f60)
+- Updated tests [`07b0843`](https://github.com/RhetTbull/osxphotos/commit/07b08433df5a60f191e23a95394e83e51dca016f)
+- Merge branch 'master' into convert_to_jpeg [`fe5185b`](https://github.com/RhetTbull/osxphotos/commit/fe5185be8893002da663039f8ec103faed0f1831)
+- Added israw, tests for Big Sur [`b5a9794`](https://github.com/RhetTbull/osxphotos/commit/b5a9794f6bff5683fd42a22197454940e4d7ba88)
+- Updates to path, path_raw, uti for RAW+JPEG pairs [`b32f4b8`](https://github.com/RhetTbull/osxphotos/commit/b32f4b8504768a5f4b5ad54c00315b9e82fca980)
+
+#### [v0.34.5](https://github.com/RhetTbull/osxphotos/compare/v0.34.3...v0.34.5)
+
+> 6 October 2020
+
+- --convert-to-jpeg initial version working [`38f201d`](https://github.com/RhetTbull/osxphotos/commit/38f201d0fb70bf299a828c1dd0d034a119e380c4)
+- Added tests, fixed bug in export_db [`5a13605`](https://github.com/RhetTbull/osxphotos/commit/5a13605f850bb947c8888246f06a5ca4e6aa5f10)
+- Updated tests [`b2b39aa`](https://github.com/RhetTbull/osxphotos/commit/b2b39aa6075df11861cf5d8945b657204f120e87)
+
+#### [v0.34.3](https://github.com/RhetTbull/osxphotos/compare/v0.34.2...v0.34.3)
+
+> 29 September 2020
+
+- Update exiftool.py to preserve file modification time, thanks to @hhoeck  [`#223`](https://github.com/RhetTbull/osxphotos/pull/223)
+- Added tests for 10.15.6 [`432da7f`](https://github.com/RhetTbull/osxphotos/commit/432da7f139a5e4b37eeb358f4ede45314407f8e5)
+- Added HEIC test image [`ddc1e69`](https://github.com/RhetTbull/osxphotos/commit/ddc1e69b4a4ac712e1af312b865c4216f9ad350c)
+- Fixed bug related to issue #222 [`c939df7`](https://github.com/RhetTbull/osxphotos/commit/c939df717159e8b97955c0b267327cd56a9ed56c)
+- Version bump for bug fix [`62d54cc`](https://github.com/RhetTbull/osxphotos/commit/62d54cc0beabd0141545608184d4b2c658eedf0f)
+- Update README.md [`6883fec`](https://github.com/RhetTbull/osxphotos/commit/6883fec2b2236d892b88327e1b4e9da1237f7dea)
+
+#### [v0.34.2](https://github.com/RhetTbull/osxphotos/compare/v0.34.1...v0.34.2)
+
+> 14 September 2020
+
+- Partial fix for issue #213 [`459d91d`](https://github.com/RhetTbull/osxphotos/commit/459d91d7b11dbd4b0564906c1689b60dc5b64642)
+
+#### [v0.34.1](https://github.com/RhetTbull/osxphotos/compare/v0.34.0...v0.34.1)
+
+> 13 September 2020
+
+- Fixed exception handling in export [`eb00ffd`](https://github.com/RhetTbull/osxphotos/commit/eb00ffd73737ef4832229e4e6fd8dc4ccb0b8539)
+- Updated README.md [`a1776fa`](https://github.com/RhetTbull/osxphotos/commit/a1776fa14850275ad6b02ece80bbe8ce908fa836)
+
+#### [v0.34.0](https://github.com/RhetTbull/osxphotos/compare/v0.33.8...v0.34.0)
+
+> 7 September 2020
+
+- Added --skip-original-if-edited for issue #159 [`5f2d401`](https://github.com/RhetTbull/osxphotos/commit/5f2d401048850fd68f31b37a7e71abc11ca80dc5)
+- Still working on issue #208 [`58b3869`](https://github.com/RhetTbull/osxphotos/commit/58b3869a7cce7cb3f211599e544d7e5426ceb4a6)
+
+#### [v0.33.8](https://github.com/RhetTbull/osxphotos/compare/v0.33.7...v0.33.8)
+
+> 31 August 2020
+
+- Fixed sidecar collisions, closes #210 [`#210`](https://github.com/RhetTbull/osxphotos/issues/210)
+
+#### [v0.33.7](https://github.com/RhetTbull/osxphotos/compare/v0.33.5...v0.33.7)
+
+> 31 August 2020
+
+- typo fix - thanks to @dmd  [`#212`](https://github.com/RhetTbull/osxphotos/pull/212)
+- Normalize unicode for issue #208 [`a36eb41`](https://github.com/RhetTbull/osxphotos/commit/a36eb416b19284477922b6a5f837f4040327138b)
+- Added force_download.py to examples [`b611d34`](https://github.com/RhetTbull/osxphotos/commit/b611d34d19db480af72f57ef55eacd0a32c8d1e8)
+- Added photoshop:SidecarForExtension to XMP, partial fix for #210 [`60d96a8`](https://github.com/RhetTbull/osxphotos/commit/60d96a8f563882fba2365a6ab58c1276725eedaa)
+- Updated README.md [`c9b1518`](https://github.com/RhetTbull/osxphotos/commit/c9b15186a022d91248451279e5f973e3f2dca4b4)
+- Update README.md [`42e8fba`](https://github.com/RhetTbull/osxphotos/commit/42e8fba125a3c6b1bd0d538f2af511aabfbeb478)
+
+#### [v0.33.5](https://github.com/RhetTbull/osxphotos/compare/v0.33.3...v0.33.5)
+
+> 25 August 2020
+
+- Fixed DST handling for from_date/to_date, closes #193 (again) [`#193`](https://github.com/RhetTbull/osxphotos/issues/193)
+- Added raw timestamps to PhotoInfo._info [`0f457a4`](https://github.com/RhetTbull/osxphotos/commit/0f457a4082a4eebc42a5df2160a02ad987b6f96c)
+
+#### [v0.33.3](https://github.com/RhetTbull/osxphotos/compare/v0.33.2...v0.33.3)
+
+> 23 August 2020
+
+- Fixed portrait for Catalina/Big Sur; see issue #203 [`1f717b0`](https://github.com/RhetTbull/osxphotos/commit/1f717b05794c2088c7c15d2aab0c5d24b6309c06)
+
+#### [v0.33.2](https://github.com/RhetTbull/osxphotos/compare/v0.33.0...v0.33.2)
+
+> 23 August 2020
+
+- Closes issue #206, adds --touch-file [`#207`](https://github.com/RhetTbull/osxphotos/pull/207)
+- Touch files - fixes #194 -- thanks to @PabloKohan  [`#205`](https://github.com/RhetTbull/osxphotos/pull/205)
+- Refactor/cleanup _export_photo - thanks to @PabloKohan  [`#204`](https://github.com/RhetTbull/osxphotos/pull/204)
+- Finished --touch-file, closes #206 [`#206`](https://github.com/RhetTbull/osxphotos/issues/206)
+- Merge pull request #205 from PabloKohan/touch_files__fix_194 [`#194`](https://github.com/RhetTbull/osxphotos/issues/194)
+- --touch-file now working with --update [`6c11e3f`](https://github.com/RhetTbull/osxphotos/commit/6c11e3fa5b5b05b98b9fdbb0e59e3a78c7dff980)
+- Refactor/cleanup _export_photo [`eefa1f1`](https://github.com/RhetTbull/osxphotos/commit/eefa1f181f4fd7b027ae69abd2b764afb590c081)
+- Fixed touch tests [`1bf7105`](https://github.com/RhetTbull/osxphotos/commit/1bf7105737fbd756064a2f9ef4d4bbd0b067978c)
+- Working on issue 206 [`ebd878a`](https://github.com/RhetTbull/osxphotos/commit/ebd878a075983ef3df0b1ead1a725e01508721f8)
+- Working on issue #206 [`c9c9202`](https://github.com/RhetTbull/osxphotos/commit/c9c920220545dc27c8cb1379d7bde15987cce72c)
+
+#### [v0.33.0](https://github.com/RhetTbull/osxphotos/compare/v0.32.0...v0.33.0)
+
+> 17 August 2020
+
+- Replaced call to which, closes #171 [`#171`](https://github.com/RhetTbull/osxphotos/issues/171)
+- Added contributors to README.md, closes #200 [`#200`](https://github.com/RhetTbull/osxphotos/issues/200)
+- Added tests for 10.15.6 [`d2deeff`](https://github.com/RhetTbull/osxphotos/commit/d2deefff834e46e1a26adc01b1b025ac839dbc78)
+- Added ImportInfo for Photos 5+ [`98e4170`](https://github.com/RhetTbull/osxphotos/commit/98e417023ec5bd8292b25040d0844f3706645950)
+- Update README.md [`360c8d8`](https://github.com/RhetTbull/osxphotos/commit/360c8d8e1b4760e95a8b71b3a0bf0df4fb5adaf5)
+- Update README.md [`868cda8`](https://github.com/RhetTbull/osxphotos/commit/868cda8482ce6b29dd00e04a209d40550e6b128b)
+
+#### [v0.32.0](https://github.com/RhetTbull/osxphotos/compare/v0.31.2...v0.32.0)
+
+> 9 August 2020
+
+- Alpha support for MacOS Big Sur/10.16, see issue #187 [`6acf9ac`](https://github.com/RhetTbull/osxphotos/commit/6acf9acd6364e1996158179493d128ec0958e652)
+
+#### [v0.31.2](https://github.com/RhetTbull/osxphotos/compare/v0.31.0...v0.31.2)
+
+> 9 August 2020
+
+- Fixed from_date and to_date to be timezone aware, closes #193 [`#193`](https://github.com/RhetTbull/osxphotos/issues/193)
+- Added test for valid XMP file, closes #197 [`#197`](https://github.com/RhetTbull/osxphotos/issues/197)
+- Dropped py36 due to datetime.fromisoformat [`a714ae0`](https://github.com/RhetTbull/osxphotos/commit/a714ae0af089b13acf70c4f29934393aa48ed222)
+- Added --uuid-from-file to CLI [`840e993`](https://github.com/RhetTbull/osxphotos/commit/840e9937bede407ef55972a361618683245e086b)
+- Added write_uuid_to_file.applescript to utils [`bea770b`](https://github.com/RhetTbull/osxphotos/commit/bea770b322d21cf3f8245d20e182006247cb71d6)
+- Updated README.md [`002fce8`](https://github.com/RhetTbull/osxphotos/commit/002fce8e93edd936d4b866118ae6d4c94e5d6744)
+- Added py37 [`d0ec862`](https://github.com/RhetTbull/osxphotos/commit/d0ec8620c721fe7576ab7d519a5eaac4d17a317e)
+
+#### [v0.31.0](https://github.com/RhetTbull/osxphotos/compare/v0.30.13...v0.31.0)
+
+> 27 July 2020
+
+- Initial FaceInfo support for Issue #21 [`6f29cda`](https://github.com/RhetTbull/osxphotos/commit/6f29cda99f1b8d94a95597c7046620cf21fecae4)
+- Updated Github Actions to run on PR [`9fc4f76`](https://github.com/RhetTbull/osxphotos/commit/9fc4f762193699dd45b586b51aa2d3066928aab1)
+
 #### [v0.30.13](https://github.com/RhetTbull/osxphotos/compare/v0.30.12...v0.30.13)
 
 > 23 July 2020

README.md

@@ -15,11 +15,13 @@
     + [PhotoInfo](#photoinfo)
     + [ExifInfo](#exifinfo)
     + [AlbumInfo](#albuminfo)
+    + [ImportInfo](#importinfo)
     + [FolderInfo](#folderinfo)
     + [PlaceInfo](#placeinfo)
     + [ScoreInfo](#scoreinfo)
     + [PersonInfo](#personinfo)
     + [FaceInfo](#faceinfo)
+    + [Raw Photos](#raw-photos)
     + [Template Substitutions](#template-substitutions)
     + [Utility Functions](#utility-functions)
   * [Examples](#examples)
@@ -52,10 +54,12 @@ OSXPhotos uses setuptools, thus simply run:
 
 	python3 setup.py install
 
-You can also install directly from [pypi](https://pypi.org/):
+You can also install directly from [pypi](https://pypi.org/project/osxphotos/):
 
     pip install osxphotos
 	
+**WARNING** The git repo for this project is very large (> 1GB) because it contains multiple Photos libraries used for testing on different versions of MacOS.  If you just want to use the osxphotos package in your own code, I recommend you install the latest version from [PyPI](https://pypi.org/project/osxphotos/). If you just want to use the command line utility, you can download a pre-built executable of the latest [release](https://github.com/RhetTbull/osxphotos/releases) or you can install via `pip` which also installs the command line app.  If you aren't comfortable with running python on your Mac, start with the pre-built executable.
+
 ## Command Line Usage
 
 This package will install a command line utility called `osxphotos` that allows you to query the Photos database.  Alternatively, you can also run the command line utility like this: `python3 -m osxphotos`
@@ -111,7 +115,7 @@ Usage: osxphotos export [OPTIONS] [PHOTOS_LIBRARY]... DEST
   for photos matching all options). If no query options are provided, all
   photos will be exported. By default, all versions of all photos will be
   exported including edited versions, live photo movies, burst photos, and
-  associated RAW images.  See --skip-edited, --skip-live, --skip-bursts, and
+  associated raw images.  See --skip-edited, --skip-live, --skip-bursts, and
   --skip-raw options to modify this behavior.
 
 Options:
@@ -196,20 +200,20 @@ Options:
   --not-selfie                    Search for photos that are not selfies.
   --panorama                      Search for panorama photos.
   --not-panorama                  Search for photos that are not panoramas.
-  --has-raw                       Search for photos with both a jpeg and RAW
+  --has-raw                       Search for photos with both a jpeg and raw
                                   version
   --only-movies                   Search only for movies (default searches
                                   both images and movies).
   --only-photos                   Search only for photos/images (default
                                   searches both images and movies).
-  --from-date [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
-                                  Search by start item date, e.g.
-                                  2000-01-12T12:00:00 or 2000-12-31 (ISO 8601
-                                  w/o TZ).
-  --to-date [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
-                                  Search by end item date, e.g.
-                                  2000-01-12T12:00:00 or 2000-12-31 (ISO 8601
-                                  w/o TZ).
+  --from-date DATETIME            Search by start item date, e.g.
+                                  2000-01-12T12:00:00,
+                                  2001-01-12T12:00:00-07:00, or 2000-12-31
+                                  (ISO 8601).
+  --to-date DATETIME              Search by end item date, e.g.
+                                  2000-01-12T12:00:00,
+                                  2001-01-12T12:00:00-07:00, or 2000-12-31
+                                  (ISO 8601).
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -221,6 +225,8 @@ Options:
   --export-as-hardlink            Hardlink files instead of copying them.
                                   Cannot be used with --exiftool which creates
                                   copies of the files with embedded EXIF data.
+  --touch-file                    Sets the file's modification time to match
+                                  photo date.
   --overwrite                     Overwrite existing files. Default behavior
                                   is to add (1), (2), etc to filename if file
                                   already exists. Use this with caution as it
@@ -231,14 +237,16 @@ Options:
                                   DEST/2019/12/20/photoname.jpg).
   --skip-edited                   Do not export edited version of photo if an
                                   edited version exists.
+  --skip-original-if-edited       Do not export original if there is an edited
+                                  version (exports only the edited version).
   --skip-bursts                   Do not export all associated burst images in
                                   the library if a photo is a burst photo.
   --skip-live                     Do not export the associated live video
                                   component of a live photo.
-  --skip-raw                      Do not export associated RAW images of a
-                                  RAW/jpeg pair.  Note: this does not skip RAW
-                                  photos if the RAW photo does not have an
-                                  associated jpeg image (e.g. the RAW file was
+  --skip-raw                      Do not export associated raw images of a
+                                  RAW+JPEG pair.  Note: this does not skip raw
+                                  photos if the raw photo does not have an
+                                  associated jpeg image (e.g. the raw file was
                                   imported to Photos without a jpeg preview).
   --person-keyword                Use person in image as keyword/tag when
                                   exporting metadata.
@@ -272,6 +280,13 @@ Options:
                                   renamed upon import.  By default, photos are
                                   exported with the the original name they had
                                   before import.
+  --convert-to-jpeg               Convert all non-jpeg images (e.g. raw, HEIC,
+                                  PNG, etc) to JPEG upon export.  Only works
+                                  if your Mac has a GPU.
+  --jpeg-quality FLOAT RANGE      Value in range 0.0 to 1.0 to use with
+                                  --convert-to-jpeg. A value of 1.0 specifies
+                                  best quality, a value of 0.0 specifies
+                                  maximum compression. Defaults to 1.0.
   --sidecar FORMAT                Create sidecar for each photo exported;
                                   valid FORMAT values: xmp, json; --sidecar
                                   json: create JSON sidecar useable by
@@ -340,7 +355,7 @@ If using --update, the exported library should be treated as a backup, not a
 working copy where you intend to make changes.
 
 Note: The number of files reported for export and the number actually exported
-may differ due to live photos, associated RAW images, and edited photos which
+may differ due to live photos, associated raw images, and edited photos which
 are reported in the total photos exported.
 
 Implementation note: To determine which files need to be updated, osxphotos
@@ -670,7 +685,7 @@ if __name__ == "__main__":
 #### Read a Photos library database
 
 ```python
-osxphotos.PhotosDB() # not recommended, see Note below
+osxphotos.PhotosDB()
 osxphotos.PhotosDB(path)
 osxphotos.PhotosDB(dbfile=path)
 ```
@@ -681,7 +696,7 @@ Pass the path to a Photos library or to a specific database file (e.g. "/Users/s
 
 If an invalid path is passed, PhotosDB will raise `FileNotFoundError` exception.
 
-**Note**: If neither path or dbfile is passed, PhotosDB will use get_last_library_path to open the last opened Photos library.  This usually works but is not 100% reliable.  It can also lead to loading a different library than expected if the user has held down *option* key when opening Photos to switch libraries.  It is therefore recommended you explicitely pass the path to `PhotosDB()`.  
+**Note**: If neither path or dbfile is passed, PhotosDB will use get_last_library_path to open the last opened Photos library.  This usually works but is not 100% reliable.  It can also lead to loading a different library than expected if the user has held down *option* key when opening Photos to switch libraries.  You may therefore want to explicitely pass the path to `PhotosDB()`. 
 
 #### Open the default (last opened) Photos library
 
@@ -767,6 +782,10 @@ Returns list of shared album names found in photos database (e.g. albums shared
 
 **Note**: *Only valid for Photos 5 / MacOS 10.15*; on Photos <= 4, prints warning and returns empty list.
 
+#### `import_info`
+
+Returns a list of [ImportInfo](#importinfo) objects representing the import sessions for the database.
+
 #### `folder_info`
 ```python
 # assumes photosdb is a PhotosDB object (see above)
@@ -1053,6 +1072,9 @@ Returns a list of albums the photo is contained in. See also [album_info](#album
 #### `album_info`
 Returns a list of [AlbumInfo](#AlbumInfo) objects representing the albums the photo is contained in.  See also [albums](#albums).
 
+#### `import_info`
+Returns an [ImportInfo](#importinfo) object representing the import session associated with the photo or `None` if there is no associated import session.
+
 #### `persons`
 Returns a list of the names of the persons in the photo
 
@@ -1070,6 +1092,18 @@ Returns the absolute path to the edited photo on disk as a string.  If the photo
 
 **Note**: will also return None if the edited photo is missing on disk. 
 
+#### `path_raw`
+Returns the absolute path to the associated raw photo on disk as a string, if photo is part of a RAW+JPEG pair, otherwise returns None.  See [notes on Raw Photos](#raw-photos).
+
+#### `has_raw`
+Returns True if photo has an associated raw image, otherwise False. (e.g. Photo is a RAW+JPEG pair). See also [is_raw](#israw) and [notes on Raw Photos](#raw-photos).
+
+#### `israw`
+Returns True if photo is a raw image. E.g. it was imported as a single raw image, not part of a RAW+JPEG pair.  See also [has_raw](#has_raw) and .
+
+#### `raw_original`
+Returns True if associated raw image and the raw image is selected in Photos via "Use RAW as Original", otherwise returns False.  See [notes on Raw Photos](#raw-photos).
+
 #### `height`
 Returns height of the photo in pixels.  If image has been edited, returns height of the edited image, otherwise returns height of the original image.  See also [original_height](#original_height).
 
@@ -1135,7 +1169,16 @@ Returns True if photo is a [cloud asset](#iscloudasset) and is synched to iCloud
 **Note**: Applies to master (original) photo only.  It's possible for the master to be in iCloud but a local edited version is not yet synched to iCloud. `incloud` provides status of only the master photo.  osxphotos does not yet provide a means to determine if the edited version is in iCloud.  If you need this feature, please open an [issue](https://github.com/RhetTbull/osxphotos/issues).
 
 #### `uti`
-Returns Uniform Type Identifier (UTI) for the image, for example: 'public.jpeg' or 'com.apple.quicktime-movie'
+Returns Uniform Type Identifier (UTI) for the current version of the image, for example: 'public.jpeg' or 'com.apple. quicktime-movie'.  If the image has been edited, `uti` will return the UTI for the edited image, otherwise it will return the UTI for the original image.
+
+#### `uti_original`
+Returns Uniform Type Identifier (UTI) for the original unedited image, for example: 'public.jpeg' or 'com.apple.quicktime-movie'.
+
+#### `uti_edited`
+Returns Uniform Type Identifier (UTI) for the edited image, for example: 'public.jpeg'.  Returns None if the photo does not have adjustments.
+
+#### `uti_raw`
+Returns Uniform Type Identifier (UTI) for the associated raw image, if there is one; for example, 'com.canon.cr2-raw-image'.  If the image is raw but not part of a RAW+JPEG pair, `uti_raw` returns None.  In this case, use `uti`, or `uti_original`.  See also [has_raw](#has_raw) and [notes on Raw Photos](#raw-photos).
 
 #### `burst`
 Returns True if photos is a burst image (e.g. part of a set of burst images), otherwise False.
@@ -1378,6 +1421,15 @@ Returns the title or name of the album.
 #### <a name="albumphotos">`photos`</a>
 Returns a list of [PhotoInfo](#PhotoInfo) objects representing each photo contained in the album sorted in the same order as in Photos. (e.g. if photos were manually sorted in the Photos albums, photos returned by `photos` will be in same order as they appear in the Photos album)
 
+#### `creation_date`
+Returns the creation date as a timezone aware datetime.datetime object of the album.
+
+#### `start_date`
+Returns the date of earliest photo in the album as a timezone aware datetime.datetime object.
+
+#### `end_date`
+Returns the date of latest photo in the album as a timezone aware datetime.datetime object.
+
 #### `folder_list`
 Returns a hierarchical list of [FolderInfo](#FolderInfo) objects representing the folders the album is contained in.  For example, if album "AlbumInFolder" is in SubFolder2 of Folder1 as illustrated below, would return a list of `FolderInfo` objects representing ["Folder1", "SubFolder2"] 
 
@@ -1403,6 +1455,25 @@ Photos Library
 #### `parent`
 Returns a [FolderInfo](#FolderInfo) object representing the albums parent folder or `None` if album is not a in a folder.
 
+### ImportInfo 
+PhotosDB.import_info returns a list of ImportInfo objects.  Each ImportInfo object represents an import session in the library.  PhotoInfo.import_info returns a single ImportInfo object representing the import session for the photo (or `None` if no associated import session).
+
+**Note**: Photos 5+ only.  Not implemented for Photos version <= 4.
+
+#### `uuid`
+Returns the universally unique identifier (uuid) of the import session.  This is how Photos keeps track of individual objects within the database.
+
+#### <a name="importphotos">`photos`</a>
+Returns a list of [PhotoInfo](#PhotoInfo) objects representing each photo contained in the import session.
+
+#### `creation_date`
+Returns the creation date as a timezone aware datetime.datetime object of the import session.
+
+#### `start_date`
+Returns the start date as a timezone aware datetime.datetime object for when the import session bega.
+
+#### `end_date`
+Returns the end date as a timezone aware datetime.datetime object for when the import session completed.
 
 ### FolderInfo 
 PhotosDB.folder_info returns a list of FolderInfo objects representing the top level folders in the library.  Each FolderInfo object represents a single folder in the Photos library.
@@ -1662,6 +1733,38 @@ Returns a dictionary representation of the FaceInfo instance.
 #### `json()`
 Returns a JSON representation of the FaceInfo instance.
 
+### Raw Photos
+Handling raw photos in `osxphotos` requires a bit of extra work.  Raw photos in Photos can be imported in two different ways: 1) a single raw photo with no associated JPEG image is imported 2) a raw+JPEG pair is imported -- two separate images with same file stem (e.g. `IMG_0001.CR2` and `IMG_001.JPG`) are imported.  
+
+The latter are treated by Photos as a single image.  By default, Photos will treat these as a JPEG image.  They are denoted in the Photos interface with a "J" icon superimposed on the image.  In Photos, the user can select "Use RAW as original" in which case the "J" icon changes to an "R" icon and all subsequent edits will use the raw image as the original. To further complicate this, different versions of Photos handle these differently in their internal logic.  
+
+`osxphotos` attempts to simplify the handling of these raw+JPEG pairs by providing a set of attributes for accessing both the JPEG and the raw version.  For example, [PhotoInfo.has_raw](#has_raw) will be True if the photo has an associated raw image but False otherwise and [PhotoInfo.path_raw](#path_raw) provides the path to the associated raw image.  Reference the following table for the various attributes useful for dealing with raw images.  Given the different ways Photos deals with raw images I've struggled with how to represent these in a logical and consistent manner.  If you have suggestions for a better interface, please open an [issue](https://github.com/RhetTbull/osxphotos/issues)!
+
+#### Raw-Related Attributes 
+
+|`PhotoInfo` attribute|`IMG_0001.CR2` imported without raw+JPEG pair|`IMG_0001.CR2` + `IMG_0001.JPG` raw+JPEG pair, JPEG is original|`IMG_0001.CR2` + `IMG_0001.JPG` raw+jpeg pair, raw is original|
+|----------|----------|----------|----------|
+|[israw](#israw)| True | False | False |
+|[has_raw](#has_raw)| False | True | True |
+|[uti](#uti) | `com.canon.cr2-raw-image` | `public.jpeg` | `public.jpeg` |
+|[uti_raw](#uti_raw) | None | `com.canon.cr2-raw-image` | `com.canon.cr2-raw-image` |
+|[raw_original](#raw_original) | False | False | True |
+|[path](#path) | `/path/to/IMG_0001.CR2` | `/path/to/IMG_0001.JPG` | `/path/to/IMG_0001.JPG` |
+|[path_raw](#path_raw) | None | `/path/to/IMG_0001.CR2` | `/path/to/IMG_0001.CR2` | 
+
+#### Example
+To get the path of every raw photo, whether it's a single raw photo or a raw+JPEG pair, one could do something like this:
+
+```python
+>>> import osxphotos
+>>> photosdb = osxphotos.PhotosDB()
+>>> photos = photosdb.photos()
+>>> all_raw = [p for p in photos if p.israw or p.has_raw]
+>>> for raw in all_raw:
+...     path = raw.path if raw.israw else raw.path_raw
+...     print(path)
+```
+
 ### Template Substitutions
 
 The following substitutions are availabe for use with `PhotoInfo.render_template()` 
@@ -1807,6 +1910,7 @@ if __name__ == "__main__":
 ## Related Projects
 
 - [rhettbull/photosmeta](https://github.com/rhettbull/photosmeta): uses osxphotos and [exiftool](https://exiftool.org/) to apply metadata from Photos as exif data in the photo files.  Can also export photos while preserving metadata and also apply Photos keywords as spotlight tags to make it easier to search for photos using spotlight.  This is mostly made obsolete by osxphotos.  The one feature that photosmeta has that osxphotos does not is ability to update the metadata of the actual photo files in the Photos library without exporting them. (Use with caution!)
+- [rhettbull/PhotoScript](https://github.com/RhetTbull/PhotoScript): python wrapper around Photos' applescript API allowing automation of Photos (including creation/deletion of items) from python.
 - [patrikhson/photo-export](https://github.com/patrikhson/photo-export): Exports older versions of Photos databases.  Provided the inspiration for osxphotos.
 - [orangeturtle739/photos-export](https://github.com/orangeturtle739/photos-export): Set of scripts to export Photos libraries.
 - [ndbroadbent/icloud_photos_downloader](https://github.com/ndbroadbent/icloud_photos_downloader): Download photos from iCloud.  Currently unmaintained.
@@ -1824,12 +1928,28 @@ If you have an interesting example that shows usage of this package, submit an i
 
 Testing against "real world" Photos libraries would be especially helpful.  If you discover issues in testing against your Photos libraries, please open an issue.  I've done extensive testing against my own Photos library but that's a since data point and I'm certain there are issues lurking in various edge cases I haven't discovered yet.
 
+### Contributors
+
+Thank-you to the following people who have contributed to improving osxphotos!  If I've inadvertently left you off, please open an issue or send me a note.
+
+- [britiscurious](https://github.com/britiscurious)
+- [Michel Wortmann](https://github.com/mwort)
+- [hshore29](https://github.com/hshore29)
+- [Pablo 'merKur' Kohan](https://github.com/PabloKohan)
+- [Jean-Yves Stervinou](https://github.com/jystervinou)
+- [Thibault Deutsch](https://github.com/dethi)
+- [grundsch](https://github.com/grundsch)
+- [Ag Primatic](https://github.com/agprimatic)
+- [Daniel M. Drucker](https://github.com/dmd)
+- [Horst Höck](https://github.com/hhoeck)
+
+
 ## Known Bugs
 
-My goal is make osxphotos as reliable and comprehensive as possible.  The test suite currently has over 600 tests--but there are still some [bugs](https://github.com/RhetTbull/osxphotos/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or incomplete features lurking.  If you find bugs please open an [issue](https://github.com/RhetTbull/osxphotos/issues).  Notable issues include:
+My goal is make osxphotos as reliable and comprehensive as possible.  The test suite currently has over 800 tests--but there are still some [bugs](https://github.com/RhetTbull/osxphotos/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or incomplete features lurking.  If you find bugs please open an [issue](https://github.com/RhetTbull/osxphotos/issues).  Please consult the list of open bugs before deciding that you want to use this code on your Photos library.  Notable issues include:
 
 - Face coordinates (mouth, left eye, right eye) may not be correct for images where the head is tilted.  See [Issue #196](https://github.com/RhetTbull/osxphotos/issues/196).
-- RAW images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the RAW image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the RAW image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
+- Raw images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the raw image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the raw image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75).
 
 ## Implementation Notes
@@ -1838,7 +1958,7 @@ This package works by creating a copy of the sqlite3 database that photos uses t
 
 If apple changes the database format this will likely break.
 
-Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the funcationality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
+Apple does provide a framework ([PhotoKit](https://developer.apple.com/documentation/photokit?language=objc)) for querying the user's Photos library and I attempted to create the functionality in this package using this framework but unfortunately PhotoKit does not provide access to much of the needed metadata (such as Faces/Persons) and Apple's System Integrity Protection (SIP) made the interface unreliable.  If you'd like to experiment with the PhotoKit interface, here's some sample [code](https://gist.github.com/RhetTbull/41cc85e5bdeb30f761147ce32fba5c94).  While copying the sqlite file is a bit kludgy, it allows osxphotos to provide access to all available metadata.
 
 For additional details about how osxphotos is implemented or if you would like to extend the code, see the [wiki](https://github.com/RhetTbull/osxphotos/wiki).
 
@@ -1849,6 +1969,8 @@ For additional details about how osxphotos is implemented or if you would like t
 - [Mako](https://www.makotemplates.org/)
 - [bpylist2](https://pypi.org/project/bpylist2/)
 - [pathvalidate](https://pypi.org/project/pathvalidate/)
+- [wurlitzer](https://pypi.org/project/wurlitzer/)
+
 
 ## Acknowledgements
 This project was originally inspired by [photo-export](https://github.com/patrikhson/photo-export) by Patrick Fältström,  Copyright (c) 2015 Patrik Fältström paf@frobbit.se

examples/force_download.py

@@ -0,0 +1,42 @@
+""" use osxphotos to force the download of photos from iCloud
+    downloads images to a temporary directory then deletes them
+    resulting in the photo being downloaded to Photos library
+"""
+
+import os
+import sys
+import tempfile
+
+import osxphotos
+
+
+def main():
+    photosdb = osxphotos.PhotosDB()
+    tempdir = tempfile.TemporaryDirectory()
+    photos = photosdb.photos()
+    downloaded = 0
+    missing = [photo for photo in photos if photo.ismissing and not photo.shared]
+
+    if not missing:
+        print(f"Did not find any missing photos to download")
+        sys.exit(0)
+
+    print(f"Downloading {len(missing)} photos")
+    for photo in missing:
+        if photo.ismissing:
+            print(f"Downloading photo {photo.original_filename}")
+            downloaded += 1
+            exported = photo.export(tempdir.name, use_photos_export=True, timeout=300)
+            if photo.hasadjustments:
+                exported.extend(
+                    photo.export(tempdir.name, use_photos_export=True, edited=True, timeout=300)
+                )
+            for filename in exported:
+                print(f"Removing temporary file {filename}")
+                os.unlink(filename)
+    print(f"Downloaded {downloaded} photos")
+    tempdir.cleanup()
+
+
+if __name__ == "__main__":
+    main()

osxphotos/__main__.py

@@ -10,6 +10,7 @@ import pathlib
 import pprint
 import sys
 import time
+import unicodedata
 
 import click
 import yaml
@@ -22,8 +23,13 @@ from pathvalidate import (
 
 import osxphotos
 
-from ._constants import _EXIF_TOOL_URL, _PHOTOS_4_VERSION, _UNKNOWN_PLACE
-from ._export_db import ExportDB, ExportDBInMemory
+from ._constants import (
+    _EXIF_TOOL_URL,
+    _PHOTOS_4_VERSION,
+    _UNKNOWN_PLACE,
+    UNICODE_FORMAT,
+)
+from .export_db import ExportDB, ExportDBInMemory
 from ._version import __version__
 from .datetime_formatter import DateTimeFormatter
 from .exiftool import get_exiftool_path
@@ -40,10 +46,24 @@ OSXPHOTOS_EXPORT_DB = ".osxphotos_export.db"
 
 
 def verbose(*args, **kwargs):
+    """ print output if verbose flag set """
     if VERBOSE:
         click.echo(*args, **kwargs)
 
 
+def normalize_unicode(value):
+    """ normalize unicode data """
+    if value is not None:
+        if isinstance(value, tuple):
+            return tuple(unicodedata.normalize(UNICODE_FORMAT, v) for v in value)
+        elif isinstance(value, str):
+            return unicodedata.normalize(UNICODE_FORMAT, value)
+        else:
+            return value
+    else:
+        return None
+
+
 def get_photos_db(*db_options):
     """ Return path to photos db, select first non-None db_options
         If no db_options are non-None, try to find library to use in
@@ -128,7 +148,7 @@ class ExportCommand(click.Command):
         formatter.write("\n")
         formatter.write_text(
             "Note: The number of files reported for export and the number actually exported "
-            + "may differ due to live photos, associated RAW images, and edited photos which are reported "
+            + "may differ due to live photos, associated raw images, and edited photos which are reported "
             + "in the total photos exported."
         )
         formatter.write("\n")
@@ -454,7 +474,7 @@ def query_options(f):
         o(
             "--has-raw",
             is_flag=True,
-            help="Search for photos with both a jpeg and RAW version",
+            help="Search for photos with both a jpeg and raw version",
         ),
         o(
             "--only-movies",
@@ -592,9 +612,9 @@ def keywords(ctx, cli_obj, db, json_, photos_library):
     photosdb = osxphotos.PhotosDB(dbfile=db)
     keywords = {"keywords": photosdb.keywords_as_dict}
     if json_ or cli_obj.json:
-        click.echo(json.dumps(keywords))
+        click.echo(json.dumps(keywords, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(keywords, sort_keys=False))
+        click.echo(yaml.dump(keywords, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -621,9 +641,9 @@ def albums(ctx, cli_obj, db, json_, photos_library):
         albums["shared albums"] = photosdb.albums_shared_as_dict
 
     if json_ or cli_obj.json:
-        click.echo(json.dumps(albums))
+        click.echo(json.dumps(albums, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(albums, sort_keys=False))
+        click.echo(yaml.dump(albums, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -647,9 +667,9 @@ def persons(ctx, cli_obj, db, json_, photos_library):
     photosdb = osxphotos.PhotosDB(dbfile=db)
     persons = {"persons": photosdb.persons_as_dict}
     if json_ or cli_obj.json:
-        click.echo(json.dumps(persons))
+        click.echo(json.dumps(persons, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(persons, sort_keys=False))
+        click.echo(yaml.dump(persons, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -673,9 +693,9 @@ def labels(ctx, cli_obj, db, json_, photos_library):
     photosdb = osxphotos.PhotosDB(dbfile=db)
     labels = {"labels": photosdb.labels_as_dict}
     if json_ or cli_obj.json:
-        click.echo(json.dumps(labels))
+        click.echo(json.dumps(labels, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(labels, sort_keys=False))
+        click.echo(yaml.dump(labels, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -733,9 +753,9 @@ def info(ctx, cli_obj, db, json_, photos_library):
     info["persons"] = persons
 
     if cli_obj.json or json_:
-        click.echo(json.dumps(info))
+        click.echo(json.dumps(info, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(info, sort_keys=False))
+        click.echo(yaml.dump(info, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -783,9 +803,9 @@ def places(ctx, cli_obj, db, json_, photos_library):
     # below needed for to make CliRunner work for testing
     cli_json = cli_obj.json if cli_obj is not None else None
     if json_ or cli_json:
-        click.echo(json.dumps(places))
+        click.echo(json.dumps(places, ensure_ascii=False))
     else:
-        click.echo(yaml.dump(places, sort_keys=False))
+        click.echo(yaml.dump(places, sort_keys=False, allow_unicode=True))
 
 
 @cli.command()
@@ -848,7 +868,7 @@ def _list_libraries(json_=False, error=True):
             "system_library": sys_lib,
             "last_library": last_lib,
         }
-        click.echo(json.dumps(libs))
+        click.echo(json.dumps(libs, ensure_ascii=False))
     else:
         last_lib_flag = sys_lib_flag = False
 
@@ -1121,6 +1141,11 @@ def query(
     help="Hardlink files instead of copying them.  "
     "Cannot be used with --exiftool which creates copies of the files with embedded EXIF data.",
 )
+@click.option(
+    "--touch-file",
+    is_flag=True,
+    help="Sets the file's modification time to match photo date.",
+)
 @click.option(
     "--overwrite",
     is_flag=True,
@@ -1140,6 +1165,11 @@ def query(
     is_flag=True,
     help="Do not export edited version of photo if an edited version exists.",
 )
+@click.option(
+    "--skip-original-if-edited",
+    is_flag=True,
+    help="Do not export original if there is an edited version (exports only the edited version).",
+)
 @click.option(
     "--skip-bursts",
     is_flag=True,
@@ -1153,9 +1183,9 @@ def query(
 @click.option(
     "--skip-raw",
     is_flag=True,
-    help="Do not export associated RAW images of a RAW/jpeg pair.  "
-    "Note: this does not skip RAW photos if the RAW photo does not have an associated jpeg image "
-    "(e.g. the RAW file was imported to Photos without a jpeg preview).",
+    help="Do not export associated raw images of a RAW+JPEG pair.  "
+    "Note: this does not skip raw photos if the raw photo does not have an associated jpeg image "
+    "(e.g. the raw file was imported to Photos without a jpeg preview).",
 )
 @click.option(
     "--person-keyword",
@@ -1200,6 +1230,21 @@ def query(
     "Note: Starting with Photos 5, all photos are renamed upon import.  By default, "
     "photos are exported with the the original name they had before import.",
 )
+@click.option(
+    "--convert-to-jpeg",
+    is_flag=True,
+    help="Convert all non-jpeg images (e.g. raw, HEIC, PNG, etc) "
+    "to JPEG upon export.  Only works if your Mac has a GPU.",
+)
+@click.option(
+    "--jpeg-quality",
+    type=click.FloatRange(0.0, 1.0),
+    default=1.0,
+    help="Value in range 0.0 to 1.0 to use with --convert-to-jpeg. " 
+    "A value of 1.0 specifies best quality, "
+    "a value of 0.0 specifies maximum compression. "
+    "Defaults to 1.0." 
+)
 @click.option(
     "--sidecar",
     default=None,
@@ -1264,6 +1309,13 @@ def query(
     "to a filesystem that doesn't support Mac OS extended attributes.  Only use this if you get "
     "an error while exporting.",
 )
+@click.option(
+    "--use-photos-export",
+    is_flag=True,
+    default=False,
+    hidden=True,
+    help="Force the use of AppleScript to export even if not missing (see also --download-missing).",
+)
 @DB_ARGUMENT
 @click.argument("dest", nargs=1, type=click.Path(exists=True))
 @click.pass_obj
@@ -1299,9 +1351,11 @@ def export(
     update,
     dry_run,
     export_as_hardlink,
+    touch_file,
     overwrite,
     export_by_date,
     skip_edited,
+    skip_original_if_edited,
     skip_bursts,
     skip_live,
     skip_raw,
@@ -1310,6 +1364,8 @@ def export(
     keyword_template,
     description_template,
     current_name,
+    convert_to_jpeg,
+    jpeg_quality,
     sidecar,
     only_photos,
     only_movies,
@@ -1344,6 +1400,7 @@ def export(
     label,
     deleted,
     deleted_only,
+    use_photos_export,
 ):
     """ Export photos from the Photos database.
         Export path DEST is required.
@@ -1352,13 +1409,13 @@ def export(
         (e.g. search for photos matching all options).
         If no query options are provided, all photos will be exported.
         By default, all versions of all photos will be exported including edited
-        versions, live photo movies, burst photos, and associated RAW images. 
+        versions, live photo movies, burst photos, and associated raw images. 
         See --skip-edited, --skip-live, --skip-bursts, and --skip-raw options
         to modify this behavior. 
     """
 
     global VERBOSE
-    VERBOSE = True if verbose_ else False
+    VERBOSE = bool(verbose_)
 
     if not os.path.isdir(dest):
         sys.exit(f"DEST {dest} must be valid path")
@@ -1383,6 +1440,8 @@ def export(
         (export_as_hardlink, exiftool),
         (any(place), no_place),
         (deleted, deleted_only),
+        (skip_edited, skip_original_if_edited),
+        (export_as_hardlink, convert_to_jpeg),
     ]
     if any(all(bb) for bb in exclusive):
         click.echo("Incompatible export options", err=True)
@@ -1450,13 +1509,22 @@ def export(
 
     if dry_run:
         export_db = ExportDBInMemory(export_db_path)
-        # echo = functools.partial(click.echo, err=True)
-        # fileutil = FileUtilNoOp(verbose=echo)
         fileutil = FileUtilNoOp
     else:
         export_db = ExportDB(export_db_path)
         fileutil = FileUtil
 
+    if verbose_:
+        if export_db.was_created:
+            verbose(f"Created export database {export_db_path}")
+        else:
+            verbose(f"Using export database {export_db_path}")
+        upgraded = export_db.was_upgraded
+        if upgraded:
+            verbose(
+                f"Upgraded export database {export_db_path} from version {upgraded[0]} to {upgraded[1]}"
+            )
+
     photos = _query(
         db=db,
         keyword=keyword,
@@ -1514,7 +1582,6 @@ def export(
         deleted_only=deleted_only,
     )
 
-    results_exported = []
     if photos:
         if export_bursts:
             # add the burst_photos to the export set
@@ -1533,10 +1600,12 @@ def export(
         # because the original code used --original-name as an option
         original_name = not current_name
 
+        results_exported = []
         results_new = []
         results_updated = []
         results_skipped = []
         results_exif_updated = []
+        results_touched = []
         if verbose_:
             for p in photos:
                 results = export_photo(
@@ -1549,6 +1618,7 @@ def export(
                     export_as_hardlink=export_as_hardlink,
                     overwrite=overwrite,
                     export_edited=export_edited,
+                    skip_original_if_edited=skip_original_if_edited,
                     original_name=original_name,
                     export_live=export_live,
                     download_missing=download_missing,
@@ -1564,13 +1634,24 @@ def export(
                     export_db=export_db,
                     fileutil=fileutil,
                     dry_run=dry_run,
+                    touch_file=touch_file,
                     edited_suffix=edited_suffix,
+                    use_photos_export=use_photos_export,
+                    convert_to_jpeg=convert_to_jpeg,
+                    jpeg_quality=jpeg_quality,
                 )
                 results_exported.extend(results.exported)
                 results_new.extend(results.new)
                 results_updated.extend(results.updated)
                 results_skipped.extend(results.skipped)
                 results_exif_updated.extend(results.exif_updated)
+                results_touched.extend(results.touched)
+
+                # if convert_to_jpeg and p.isphoto and p.uti != "public.jpeg":
+                #     for photo_file in set(
+                #         results.exported + results.updated + results.exif_updated
+                #     ):
+                #         verbose(f"Converting {photo_file} to jpeg")
 
         else:
             # show progress bar
@@ -1586,6 +1667,7 @@ def export(
                         export_as_hardlink=export_as_hardlink,
                         overwrite=overwrite,
                         export_edited=export_edited,
+                        skip_original_if_edited=skip_original_if_edited,
                         original_name=original_name,
                         export_live=export_live,
                         download_missing=download_missing,
@@ -1601,31 +1683,41 @@ def export(
                         export_db=export_db,
                         fileutil=fileutil,
                         dry_run=dry_run,
+                        touch_file=touch_file,
                         edited_suffix=edited_suffix,
+                        use_photos_export=use_photos_export,
+                        convert_to_jpeg=convert_to_jpeg,
+                        jpeg_quality=jpeg_quality,
                     )
                     results_exported.extend(results.exported)
                     results_new.extend(results.new)
                     results_updated.extend(results.updated)
                     results_skipped.extend(results.skipped)
                     results_exif_updated.extend(results.exif_updated)
+                    results_touched.extend(results.touched)
+
         stop_time = time.perf_counter()
         # print summary results
         if update:
             photo_str_new = "photos" if len(results_new) != 1 else "photo"
-            photo_str_updated = "photos" if len(results_new) != 1 else "photo"
+            photo_str_updated = "photos" if len(results_updated) != 1 else "photo"
             photo_str_skipped = "photos" if len(results_skipped) != 1 else "photo"
             photo_str_exif_updated = (
                 "photos" if len(results_exif_updated) != 1 else "photo"
             )
-            click.echo(
+            summary = (
                 f"Exported: {len(results_new)} {photo_str_new}, "
-                + f"updated: {len(results_updated)} {photo_str_updated}, "
-                + f"skipped: {len(results_skipped)} {photo_str_skipped}, "
-                + f"updated EXIF data: {len(results_exif_updated)} {photo_str_exif_updated}"
+                f"updated: {len(results_updated)} {photo_str_updated}, "
+                f"skipped: {len(results_skipped)} {photo_str_skipped}, "
+                f"updated EXIF data: {len(results_exif_updated)} {photo_str_exif_updated}"
             )
         else:
             photo_str = "photos" if len(results_exported) != 1 else "photo"
-            click.echo(f"Exported: {len(results_exported)} {photo_str}")
+            summary = f"Exported: {len(results_exported)} {photo_str}"
+        photo_str_touched = "photos" if len(results_touched) != 1 else "photo"
+        if touch_file:
+            summary += f", touched date: {len(results_touched)} {photo_str_touched}"
+        click.echo(summary)
         click.echo(f"Elapsed time: {(stop_time-start_time):.3f} seconds")
     else:
         click.echo("Did not find any photos to export")
@@ -1834,6 +1926,15 @@ def _query(
             to_date=to_date,
         )
 
+    person = normalize_unicode(person)
+    keyword = normalize_unicode(keyword)
+    album = normalize_unicode(album)
+    folder = normalize_unicode(folder)
+    title = normalize_unicode(title)
+    description = normalize_unicode(description)
+    place = normalize_unicode(place)
+    label = normalize_unicode(label)
+
     if album:
         photos = get_photos_by_attribute(photos, "albums", album, ignore_case)
 
@@ -1957,7 +2058,7 @@ def _query(
         photos = [p for p in photos if not p.shared]
 
     if uti:
-        photos = [p for p in photos if uti in p.uti]
+        photos = [p for p in photos if uti in p.uti_original]
 
     if burst:
         photos = [p for p in photos if p.burst]
@@ -2058,6 +2159,7 @@ def export_photo(
     export_as_hardlink=None,
     overwrite=None,
     export_edited=None,
+    skip_original_if_edited=None,
     original_name=None,
     export_live=None,
     download_missing=None,
@@ -2073,7 +2175,11 @@ def export_photo(
     export_db=None,
     fileutil=FileUtil,
     dry_run=None,
+    touch_file=None,
     edited_suffix="_edited",
+    use_photos_export=False,
+    convert_to_jpeg=False,
+    jpeg_quality=1.0,
 ):
     """ Helper function for export that does the actual export
 
@@ -2093,7 +2199,9 @@ def export_photo(
         directory: template used to determine output directory
         filename_template: template use to determine output file
         no_extended_attributes: boolean; if True, exports photo without preserving extended attributes
-        export_raw: boolean; if True exports RAW image associate with the photo
+        export_raw: boolean; if True exports raw image associate with the photo
+        export_edited: boolean; if True exports edited version of photo if there is one
+        skip_original_if_edited: boolean; if True does not export original if photo has been edited
         album_keyword: boolean; if True, exports album names as keywords in metadata
         person_keyword: boolean; if True, exports person names as keywords in metadata
         keyword_template: list of strings; if provided use rendered template strings as keywords
@@ -2101,6 +2209,10 @@ def export_photo(
         export_db: export database instance compatible with ExportDB_ABC
         fileutil: file util class compatible with FileUtilABC
         dry_run: boolean; if True, doesn't actually export or update any files
+        touch_file: boolean; sets file's modification time to match photo date
+        use_photos_export: boolean; if True forces the use of AppleScript to export even if photo not missing
+        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
     Returns:
         list of path(s) of exported photo or None if photo was missing
@@ -2109,35 +2221,38 @@ def export_photo(
         ValueError on invalid filename_template
     """
     global VERBOSE
-    VERBOSE = True if verbose_ else False
+    VERBOSE = bool(verbose_)
 
     if not download_missing:
         if photo.ismissing:
             space = " " if not verbose_ else ""
-            verbose(f"{space}Skipping missing photo {photo.filename}")
-            return ExportResults([], [], [], [], [])
+            verbose(f"{space}Skipping missing photo {photo.original_filename}")
+            return ExportResults([], [], [], [], [], [])
         elif not os.path.exists(photo.path):
             space = " " if not verbose_ else ""
             verbose(
                 f"{space}WARNING: file {photo.path} is missing but ismissing=False, "
-                f"skipping {photo.filename}"
+                f"skipping {photo.original_filename}"
             )
-            return ExportResults([], [], [], [], [])
+            return ExportResults([], [], [], [], [], [])
     elif photo.ismissing and not photo.iscloudasset or not photo.incloud:
         verbose(
-            f"Skipping missing {photo.filename}: not iCloud asset or missing from cloud"
+            f"Skipping missing {photo.original_filename}: not iCloud asset or missing from cloud"
         )
-        return ExportResults([], [], [], [], [])
+        return ExportResults([], [], [], [], [], [])
 
     results_exported = []
     results_new = []
     results_updated = []
     results_skipped = []
     results_exif_updated = []
+    results_touched = []
+
+    export_original = not (skip_original_if_edited and photo.hasadjustments)
 
     filenames = get_filenames_from_template(photo, filename_template, original_name)
     for filename in filenames:
-        verbose(f"Exporting {photo.filename} as {filename}")
+        verbose(f"Exporting {photo.original_filename} ({photo.filename}) as {filename}")
 
         dest_paths = get_dirnames_from_template(
             photo, directory, export_by_date, dest, dry_run
@@ -2152,56 +2267,66 @@ def export_photo(
 
         # if download_missing and the photo is missing or path doesn't exist,
         # try to download with Photos
-        use_photos_export = download_missing and (
-            photo.ismissing or not os.path.exists(photo.path)
+        use_photos_export = (
+            download_missing and (photo.ismissing or not os.path.exists(photo.path))
+            if not use_photos_export
+            else True
         )
 
         # export the photo to each path in dest_paths
         for dest_path in dest_paths:
-            export_results = photo.export2(
-                dest_path,
-                filename,
-                sidecar_json=sidecar_json,
-                sidecar_xmp=sidecar_xmp,
-                live_photo=export_live,
-                raw_photo=export_raw,
-                export_as_hardlink=export_as_hardlink,
-                overwrite=overwrite,
-                use_photos_export=use_photos_export,
-                exiftool=exiftool,
-                no_xattr=no_extended_attributes,
-                use_albums_as_keywords=album_keyword,
-                use_persons_as_keywords=person_keyword,
-                keyword_template=keyword_template,
-                description_template=description_template,
-                update=update,
-                export_db=export_db,
-                fileutil=fileutil,
-                dry_run=dry_run,
-            )
+            if not export_original:
+                verbose(f"Skipping original version of {photo.original_filename}")
+            else:
+                export_results = photo.export2(
+                    dest_path,
+                    filename,
+                    sidecar_json=sidecar_json,
+                    sidecar_xmp=sidecar_xmp,
+                    live_photo=export_live,
+                    raw_photo=export_raw,
+                    export_as_hardlink=export_as_hardlink,
+                    overwrite=overwrite,
+                    use_photos_export=use_photos_export,
+                    exiftool=exiftool,
+                    no_xattr=no_extended_attributes,
+                    use_albums_as_keywords=album_keyword,
+                    use_persons_as_keywords=person_keyword,
+                    keyword_template=keyword_template,
+                    description_template=description_template,
+                    update=update,
+                    export_db=export_db,
+                    fileutil=fileutil,
+                    dry_run=dry_run,
+                    touch_file=touch_file,
+                    convert_to_jpeg=convert_to_jpeg,
+                    jpeg_quality=jpeg_quality,
+                )
 
-            results_exported.extend(export_results.exported)
-            results_new.extend(export_results.new)
-            results_updated.extend(export_results.updated)
-            results_skipped.extend(export_results.skipped)
-            results_exif_updated.extend(export_results.exif_updated)
+                results_exported.extend(export_results.exported)
+                results_new.extend(export_results.new)
+                results_updated.extend(export_results.updated)
+                results_skipped.extend(export_results.skipped)
+                results_exif_updated.extend(export_results.exif_updated)
+                results_touched.extend(export_results.touched)
 
-            if verbose_:
-                for exported in export_results.exported:
-                    verbose(f"Exported {exported}")
-                for new in export_results.new:
-                    verbose(f"Exported new file {new}")
-                for updated in export_results.updated:
-                    verbose(f"Exported updated file {updated}")
-                for skipped in export_results.skipped:
-                    verbose(f"Skipped up to date file {skipped}")
+                if verbose_:
+                    for exported in export_results.exported:
+                        verbose(f"Exported {exported}")
+                    for new in export_results.new:
+                        verbose(f"Exported new file {new}")
+                    for updated in export_results.updated:
+                        verbose(f"Exported updated file {updated}")
+                    for skipped in export_results.skipped:
+                        verbose(f"Skipped up to date file {skipped}")
+                    for touched in export_results.touched:
+                        verbose(f"Touched date on file {touched}")
 
             # if export-edited, also export the edited version
             # verify the photo has adjustments and valid path to avoid raising an exception
             if export_edited and photo.hasadjustments:
                 # if download_missing and the photo is missing or path doesn't exist,
                 # try to download with Photos
-                use_photos_export = download_missing and photo.path_edited is None
                 if not download_missing and photo.path_edited is None:
                     verbose(f"Skipping missing edited photo for {filename}")
                 else:
@@ -2234,6 +2359,9 @@ def export_photo(
                         export_db=export_db,
                         fileutil=fileutil,
                         dry_run=dry_run,
+                        touch_file=touch_file,
+                        convert_to_jpeg=convert_to_jpeg,
+                        jpeg_quality=jpeg_quality,
                     )
 
                     results_exported.extend(export_results_edited.exported)
@@ -2241,6 +2369,7 @@ def export_photo(
                     results_updated.extend(export_results_edited.updated)
                     results_skipped.extend(export_results_edited.skipped)
                     results_exif_updated.extend(export_results_edited.exif_updated)
+                    results_touched.extend(export_results_edited.touched)
 
                     if verbose_:
                         for exported in export_results_edited.exported:
@@ -2251,6 +2380,8 @@ def export_photo(
                             verbose(f"Exported updated file {updated}")
                         for skipped in export_results_edited.skipped:
                             verbose(f"Skipped up to date file {skipped}")
+                        for touched in export_results_edited.touched:
+                            verbose(f"Touched date on file {touched}")
 
     return ExportResults(
         results_exported,
@@ -2258,6 +2389,7 @@ def export_photo(
         results_updated,
         results_skipped,
         results_exif_updated,
+        results_touched,
     )
 
 

osxphotos/_constants.py

@@ -3,6 +3,14 @@ Constants used by osxphotos
 """
 
 import os.path
+from datetime import datetime
+
+# Time delta: add this to Photos times to get unix time
+# Apple Epoch is Jan 1, 2001
+TIME_DELTA = (datetime(2001, 1, 1, 0, 0) - datetime(1970, 1, 1, 0, 0)).total_seconds()
+
+# Unicode format to use for comparing strings
+UNICODE_FORMAT = "NFC"
 
 # which Photos library database versions have been tested
 # Photos 2.0 (10.12.6) == 2622
@@ -36,11 +44,17 @@ _DB_TABLE_NAMES = {
         "ASSET": "ZGENERICASSET",
         "KEYWORD_JOIN": "Z_1KEYWORDS.Z_37KEYWORDS",
         "ALBUM_JOIN": "Z_26ASSETS.Z_34ASSETS",
+        "ALBUM_SORT_ORDER": "Z_26ASSETS.Z_FOK_34ASSETS",
+        "IMPORT_FOK": "ZGENERICASSET.Z_FOK_IMPORTSESSION",
+        "DEPTH_STATE": "ZGENERICASSET.ZDEPTHSTATES",
     },
     6: {
         "ASSET": "ZASSET",
         "KEYWORD_JOIN": "Z_1KEYWORDS.Z_36KEYWORDS",
         "ALBUM_JOIN": "Z_26ASSETS.Z_3ASSETS",
+        "ALBUM_SORT_ORDER": "Z_26ASSETS.Z_FOK_3ASSETS",
+        "IMPORT_FOK": "null",
+        "DEPTH_STATE": "ZASSET.ZDEPTHTYPE",
     },
 }
 
@@ -71,6 +85,7 @@ _PHOTOS_5_ALBUM_KIND = 2  # normal user album
 _PHOTOS_5_SHARED_ALBUM_KIND = 1505  # shared album
 _PHOTOS_5_FOLDER_KIND = 4000  # user folder
 _PHOTOS_5_ROOT_FOLDER_KIND = 3999  # root folder
+_PHOTOS_5_IMPORT_SESSION_ALBUM_KIND = 1506  # import session
 
 _PHOTOS_4_ALBUM_KIND = 3  # RKAlbum.albumSubclass
 _PHOTOS_4_TOP_LEVEL_ALBUM = "TopLevelAlbums"

osxphotos/_version.py

@@ -1,3 +1,4 @@
 """ version info """
 
-__version__ = "0.32.0"
+__version__ = "0.35.2"
+

osxphotos/albuminfo.py

@@ -10,7 +10,7 @@ Represents a single Folder in the Photos library and provides access to the fold
 PhotosDB.folders() returns a list of FolderInfo objects
 """
 
-import logging
+from datetime import datetime, timedelta, timezone
 
 from ._constants import (
     _PHOTOS_4_ALBUM_KIND,
@@ -18,11 +18,34 @@ from ._constants import (
     _PHOTOS_4_VERSION,
     _PHOTOS_5_ALBUM_KIND,
     _PHOTOS_5_FOLDER_KIND,
+    TIME_DELTA,
 )
+from .datetime_utils import get_local_tz
 
 
-class AlbumInfo:
+def sort_list_by_keys(values, sort_keys):
+    """ Sorts list values by a second list sort_keys
+        e.g. given ["a","c","b"], [1, 3, 2], returns ["a", "b", "c"]
+
+    Args:
+        values: a list of values to be sorted
+        sort_keys: a list of keys to sort values by
+    
+    Returns:
+        list of values, sorted by sort_keys
+    
+    Raises:
+        ValueError: raised if len(values) != len(sort_keys)
     """
+    if len(values) != len(sort_keys):
+        return ValueError("values and sort_keys must have same length")
+
+    return list(zip(*sorted(zip(sort_keys, values))))[1]
+
+
+class AlbumInfoBaseClass:
+    """
+    Base class for AlbumInfo, ImportInfo
     Info about a specific Album, contains all the details about the album
     including folders, photos, etc.
     """
@@ -31,33 +54,109 @@ class AlbumInfo:
         self._uuid = uuid
         self._db = db
         self._title = self._db._dbalbum_details[uuid]["title"]
-
-    @property
-    def title(self):
-        """ return title / name of album """
-        return self._title
+        self._creation_date_timestamp = self._db._dbalbum_details[uuid]["creation_date"]
+        self._start_date_timestamp = self._db._dbalbum_details[uuid]["start_date"]
+        self._end_date_timestamp = self._db._dbalbum_details[uuid]["end_date"]
+        self._local_tz = get_local_tz(
+            datetime.fromtimestamp(self._creation_date_timestamp + TIME_DELTA)
+        )
 
     @property
     def uuid(self):
         """ return uuid of album """
         return self._uuid
 
+    @property
+    def creation_date(self):
+        """ return creation date of album """
+        try:
+            return self._creation_date
+        except AttributeError:
+            try:
+                self._creation_date = (
+                    datetime.fromtimestamp(
+                        self._creation_date_timestamp + TIME_DELTA
+                    ).astimezone(tz=self._local_tz)
+                    if self._creation_date_timestamp
+                    else datetime(1970, 1, 1, 0, 0, 0).astimezone(
+                        tz=timezone(timedelta(0))
+                    )
+                )
+            except ValueError:
+                self._creation_date = datetime(1970, 1, 1, 0, 0, 0).astimezone(
+                    tz=timezone(timedelta(0))
+                )
+            return self._creation_date
+
+    @property
+    def start_date(self):
+        """ For Albums, return start date (earliest image) of album or None for albums with no images
+            For Import Sessions, return start date of import session (when import began) """
+        try:
+            return self._start_date
+        except AttributeError:
+            try:
+                self._start_date = (
+                    datetime.fromtimestamp(
+                        self._start_date_timestamp + TIME_DELTA
+                    ).astimezone(tz=self._local_tz)
+                    if self._start_date_timestamp
+                    else None
+                )
+            except ValueError:
+                self._start_date = None
+            return self._start_date
+
+    @property
+    def end_date(self):
+        """ For Albums, return end date (most recent image) of album or None for albums with no images
+            For Import Sessions, return end date of import sessions (when import was completed) """
+        try:
+            return self._end_date
+        except AttributeError:
+            try:
+                self._end_date = (
+                    datetime.fromtimestamp(
+                        self._end_date_timestamp + TIME_DELTA
+                    ).astimezone(tz=self._local_tz)
+                    if self._end_date_timestamp
+                    else None
+                )
+            except ValueError:
+                self._end_date = None
+            return self._end_date
+
     @property
     def photos(self):
-        """ return list of photos contained in album """
+        return []
+
+    def __len__(self):
+        """ return number of photos contained in album """
+        return len(self.photos)
+
+
+class AlbumInfo(AlbumInfoBaseClass):
+    """
+    Base class for AlbumInfo, ImportInfo
+    Info about a specific Album, contains all the details about the album
+    including folders, photos, etc.
+    """
+
+    @property
+    def title(self):
+        """ return title / name of album """
+        return self._title
+
+    @property
+    def photos(self):
+        """ return list of photos contained in album sorted in same sort order as Photos """
         try:
             return self._photos
         except AttributeError:
             if self.uuid in self._db._dbalbums_album:
                 uuid, sort_order = zip(*self._db._dbalbums_album[self.uuid])
-                self._photos = self._db.photos(uuid=uuid)
-                # PhotosDB.photos does not preserve order when passing in list of uuids
-                # so need to build photo list one a time
-                # sort uuids by sort order
-                sorted_uuid = sorted(zip(sort_order, uuid))
-                self._photos = [
-                    self._db.photos(uuid=[uuid])[0] for _, uuid in sorted_uuid
-                ]
+                sorted_uuid = sort_list_by_keys(uuid, sort_order)
+                self._photos = self._db.photos_by_uuid(sorted_uuid)
             else:
                 self._photos = []
             return self._photos
@@ -110,9 +209,24 @@ class AlbumInfo:
                 )
             return self._parent
 
-    def __len__(self):
-        """ return number of photos contained in album """
-        return len(self.photos)
+
+class ImportInfo(AlbumInfoBaseClass):
+    @property
+    def photos(self):
+        """ return list of photos contained in import session """
+        try:
+            return self._photos
+        except AttributeError:
+            uuid_list, sort_order = zip(
+                *[
+                    (uuid, self._db._dbphotos[uuid]["fok_import_session"])
+                    for uuid in self._db._dbphotos
+                    if self._db._dbphotos[uuid]["import_uuid"] == self.uuid
+                ]
+            )
+            sorted_uuid = sort_list_by_keys(uuid_list, sort_order)
+            self._photos = self._db.photos_by_uuid(sorted_uuid)
+            return self._photos
 
 
 class FolderInfo:

osxphotos/datetime_utils.py

@@ -2,14 +2,23 @@
 
 import datetime
 
-def get_local_tz():
-    """ return local timezone as datetime.timezone tzinfo """
-    local_tz = (
-        datetime.datetime.now(datetime.timezone(datetime.timedelta(0)))
-        .astimezone()
-        .tzinfo
-    )
-    return local_tz
+
+def get_local_tz(dt):
+    """ return local timezone as datetime.timezone tzinfo for dt
+    
+    Args:
+        dt: datetime.datetime
+    
+    Returns:
+        local timezone for dt as datetime.timezone
+
+    Raises:
+        ValueError if dt is not timezone naive
+    """
+    if not datetime_has_tz(dt):
+        return dt.astimezone().tzinfo
+    else:
+        raise ValueError("dt must be naive datetime.datetime object")
 
 
 def datetime_remove_tz(dt):
@@ -20,8 +29,7 @@ def datetime_remove_tz(dt):
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
 
-    dt_new = dt.replace(tzinfo=None)
-    return dt_new
+    return dt.replace(tzinfo=None)
 
 
 def datetime_has_tz(dt):
@@ -32,9 +40,7 @@ def datetime_has_tz(dt):
     if type(dt) != datetime.datetime:
         raise TypeError(f"dt must be type datetime.datetime, not {type(dt)}")
 
-    if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
-        return True
-    return False
+    return dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None
 
 
 def datetime_naive_to_local(dt):
@@ -53,5 +59,4 @@ def datetime_naive_to_local(dt):
             f"{dt} has tzinfo {dt.tzinfo} and offset {dt.tizinfo.utcoffset(dt)}"
         )
 
-    dt_local = dt.replace(tzinfo=get_local_tz())
-    return dt_local
+    return dt.replace(tzinfo=get_local_tz(dt))

osxphotos/exiftool.py

@@ -8,6 +8,7 @@
 import json
 import logging
 import os
+import shutil
 import subprocess
 import sys
 from functools import lru_cache  # pylint: disable=syntax-error
@@ -22,8 +23,7 @@ EXIFTOOL_STAYOPEN_EOF_LEN = len(EXIFTOOL_STAYOPEN_EOF)
 @lru_cache(maxsize=1)
 def get_exiftool_path():
     """ return path of exiftool, cache result """
-    result = subprocess.run(["which", "exiftool"], stdout=subprocess.PIPE)
-    exiftool_path = result.stdout.decode("utf-8")
+    exiftool_path = shutil.which('exiftool')
     if _debug():
         logging.debug("exiftool path = %s" % (exiftool_path))
     if exiftool_path:
@@ -98,6 +98,7 @@ class _ExifToolProc:
                 "-",  # read from stdin
                 "-common_args",  # specifies args common to all commands subsequently run
                 "-n",  # no print conversion (e.g. print tag values in machine readable format)
+                "-P",  # Preserve file modification date/time (possible interfere w/ --touch-file)
                 "-G",  # print group name for each tag
             ],
             stdin=subprocess.PIPE,

osxphotos/export_db.py

@@ -14,7 +14,7 @@ from sqlite3 import Error
 
 from ._version import __version__
 
-OSXPHOTOS_EXPORTDB_VERSION = "1.0"
+OSXPHOTOS_EXPORTDB_VERSION = "2.0"
 
 
 class ExportDB_ABC(ABC):
@@ -36,6 +36,22 @@ class ExportDB_ABC(ABC):
     def get_stat_orig_for_file(self, filename):
         pass
 
+    @abstractmethod
+    def set_stat_edited_for_file(self, filename, stats):
+        pass
+
+    @abstractmethod
+    def get_stat_edited_for_file(self, filename):
+        pass
+
+    @abstractmethod
+    def set_stat_converted_for_file(self, filename, stats):
+        pass
+
+    @abstractmethod
+    def get_stat_converted_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_stat_exif_for_file(self, filename, stats):
         pass
@@ -61,13 +77,28 @@ class ExportDB_ABC(ABC):
         pass
 
     @abstractmethod
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         pass
 
 
 class ExportDBNoOp(ExportDB_ABC):
     """ An ExportDB with NoOp methods """
 
+    def __init__(self):
+        self.was_created = True
+        self.was_upgraded = False
+        self.version = OSXPHOTOS_EXPORTDB_VERSION
+
     def get_uuid_for_file(self, filename):
         pass
 
@@ -80,6 +111,18 @@ class ExportDBNoOp(ExportDB_ABC):
     def get_stat_orig_for_file(self, filename):
         pass
 
+    def set_stat_edited_for_file(self, filename, stats):
+        pass
+
+    def get_stat_edited_for_file(self, filename):
+        pass
+
+    def set_stat_converted_for_file(self, filename, stats):
+        pass
+
+    def get_stat_converted_for_file(self, filename):
+        pass
+
     def set_stat_exif_for_file(self, filename, stats):
         pass
 
@@ -98,7 +141,17 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         pass
 
 
@@ -122,7 +175,6 @@ class ExportDB(ExportDB_ABC):
             returns None if filename not found in database
         """
         filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
-        logging.debug(f"get_uuid: {filename}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -135,14 +187,12 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             uuid = None
 
-        logging.debug(f"get_uuid: {uuid}")
         return uuid
 
     def set_uuid_for_file(self, filename, uuid):
         """ set UUID of filename to uuid in the database """
         filename = str(pathlib.Path(filename).relative_to(self._path))
         filename_normalized = filename.lower()
-        logging.debug(f"set_uuid: {filename} {uuid}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -162,7 +212,6 @@ class ExportDB(ExportDB_ABC):
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
-        logging.debug(f"set_stat_orig_for_file: {filename} {stats}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -189,14 +238,30 @@ class ExportDB(ExportDB_ABC):
                 (filename,),
             )
             results = c.fetchone()
-            stats = results[0:3] if results else None
+            if results:
+                stats = results[0:3]
+                mtime = int(stats[2]) if stats[2] is not None else None
+                stats = (stats[0], stats[1], mtime)
+            else:
+                stats = (None, None, None)
         except Error as e:
             logging.warning(e)
             stats = (None, None, None)
 
-        logging.debug(f"get_stat_orig_for_file: {stats}")
         return stats
 
+    def set_stat_edited_for_file(self, filename, stats):
+        """ set stat info for edited version of image (in Photos' library)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._set_stat_for_file("edited", filename, stats)
+
+    def get_stat_edited_for_file(self, filename):
+        """ get stat info for edited version of image (in Photos' library)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._get_stat_for_file("edited", filename)
+
     def set_stat_exif_for_file(self, filename, stats):
         """ set stat info for filename (after exiftool has updated it)
             filename: filename to set the stat info for
@@ -205,7 +270,6 @@ class ExportDB(ExportDB_ABC):
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
-        logging.debug(f"set_stat_exif_for_file: {filename} {stats}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -232,14 +296,30 @@ class ExportDB(ExportDB_ABC):
                 (filename,),
             )
             results = c.fetchone()
-            stats = results[0:3] if results else None
+            if results:
+                stats = results[0:3]
+                mtime = int(stats[2]) if stats[2] is not None else None
+                stats = (stats[0], stats[1], mtime)
+            else:
+                stats = (None, None, None)
         except Error as e:
             logging.warning(e)
             stats = (None, None, None)
 
-        logging.debug(f"get_stat_exif_for_file: {stats}")
         return stats
 
+    def set_stat_converted_for_file(self, filename, stats):
+        """ set stat info for filename (after image converted to jpeg)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._set_stat_for_file("converted", filename, stats)
+
+    def get_stat_converted_for_file(self, filename):
+        """ get stat info for filename (after jpeg conversion)
+            returns: tuple of (mode, size, mtime) 
+        """
+        return self._get_stat_for_file("converted", filename)
+
     def get_info_for_uuid(self, uuid):
         """ returns the info JSON struct for a UUID """
         conn = self._conn
@@ -252,7 +332,6 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             info = None
 
-        logging.debug(f"get_info: {uuid}, {info}")
         return info
 
     def set_info_for_uuid(self, uuid, info):
@@ -268,8 +347,6 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
-        logging.debug(f"set_info: {uuid}, {info}")
-
     def get_exifdata_for_file(self, filename):
         """ returns the exifdata JSON struct for a file """
         filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
@@ -286,7 +363,6 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             exifdata = None
 
-        logging.debug(f"get_exifdata: {filename}, {exifdata}")
         return exifdata
 
     def set_exifdata_for_file(self, filename, exifdata):
@@ -303,9 +379,17 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
-        logging.debug(f"set_exifdata: {filename}, {exifdata}")
-
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         """ sets all the data for file and uuid at once 
         """
         filename = str(pathlib.Path(filename).relative_to(self._path))
@@ -329,6 +413,14 @@ class ExportDB(ExportDB_ABC):
                 + "WHERE filepath_normalized = ?;",
                 (*exif_stat, filename_normalized),
             )
+            c.execute(
+                "INSERT OR REPLACE INTO converted(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                (filename_normalized, *converted_stat),
+            )
+            c.execute(
+                "INSERT OR REPLACE INTO edited(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                (filename_normalized, *edited_stat),
+            )
             c.execute(
                 "INSERT OR REPLACE INTO info(uuid, json_info) VALUES (?, ?);",
                 (uuid, info_json),
@@ -348,6 +440,37 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def _set_stat_for_file(self, table, filename, stats):
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        if len(stats) != 3:
+            raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
+
+        conn = self._conn
+        c = conn.cursor()
+        c.execute(
+            f"INSERT OR REPLACE INTO {table}(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+            (filename, *stats),
+        )
+        conn.commit()
+
+    def _get_stat_for_file(self, table, filename):
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        c = conn.cursor()
+        c.execute(
+            f"SELECT mode, size, mtime FROM {table} WHERE filepath_normalized = ?",
+            (filename,),
+        )
+        results = c.fetchone()
+        if results:
+            stats = results[0:3]
+            mtime = int(stats[2]) if stats[2] is not None else None
+            stats = (stats[0], stats[1], mtime)
+        else:
+            stats = (None, None, None)
+
+        return stats
+
     def _open_export_db(self, dbfile):
         """ open export database and return a db connection
             if dbfile does not exist, will create and initialize the database 
@@ -355,15 +478,24 @@ class ExportDB(ExportDB_ABC):
         """
 
         if not os.path.isfile(dbfile):
-            logging.debug(f"dbfile {dbfile} doesn't exist, creating it")
             conn = self._get_db_connection(dbfile)
             if conn:
                 self._create_db_tables(conn)
+                self.was_created = True
+                self.was_upgraded = ()
+                self.version = OSXPHOTOS_EXPORTDB_VERSION
             else:
                 raise Exception("Error getting connection to database {dbfile}")
         else:
-            logging.debug(f"dbfile {dbfile} exists, opening it")
             conn = self._get_db_connection(dbfile)
+            self.was_created = False
+            version_info = self._get_database_version(conn)
+            if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION:
+                self._create_db_tables(conn)
+                self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
+            else:
+                self.was_upgraded = ()
+            self.version = OSXPHOTOS_EXPORTDB_VERSION
 
         return conn
 
@@ -377,6 +509,13 @@ class ExportDB(ExportDB_ABC):
 
         return conn
 
+    def _get_database_version(self, conn):
+        """ return tuple of (osxphotos, exportdb) versions for database connection conn """
+        version_info = conn.execute(
+            "SELECT osxphotos, exportdb, max(id) FROM version"
+        ).fetchone()
+        return (version_info[0], version_info[1])
+
     def _create_db_tables(self, conn):
         """ create (if not already created) the necessary db tables for the export database
             conn: sqlite3 db connection 
@@ -417,9 +556,25 @@ class ExportDB(ExportDB_ABC):
                              filepath_normalized TEXT NOT NULL,
                              json_exifdata JSON 
                              ); """,
-            "sql_files_idx": """ CREATE UNIQUE INDEX idx_files_filepath_normalized on files (filepath_normalized); """,
-            "sql_info_idx": """ CREATE UNIQUE INDEX idx_info_uuid on info (uuid); """,
-            "sql_exifdata_idx": """ CREATE UNIQUE INDEX idx_exifdata_filename on exifdata (filepath_normalized); """,
+            "sql_edited_table": """ CREATE TABLE IF NOT EXISTS edited (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
+            "sql_converted_table": """ CREATE TABLE IF NOT EXISTS converted (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
+            "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
+            "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
+            "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
+            "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
+            "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
         }
         try:
             c = conn.cursor()
@@ -435,11 +590,10 @@ class ExportDB(ExportDB_ABC):
 
     def __del__(self):
         """ ensure the database connection is closed """
-        if self._conn:
-            try:
-                self._conn.close()
-            except Error as e:
-                logging.warning(e)
+        try:
+            self._conn.close()
+        except:
+            pass
 
     def _insert_run_info(self):
         dt = datetime.datetime.utcnow().isoformat()
@@ -478,18 +632,18 @@ class ExportDBInMemory(ExportDB):
 
     def _open_export_db(self, dbfile):
         """ open export database and return a db connection
-            if dbfile does not exist, will create and initialize the database 
             returns: connection to the database 
         """
         if not os.path.isfile(dbfile):
-            logging.debug(f"dbfile {dbfile} doesn't exist, creating in memory version")
             conn = self._get_db_connection()
             if conn:
                 self._create_db_tables(conn)
+                self.was_created = True
+                self.was_upgraded = ()
+                self.version = OSXPHOTOS_EXPORTDB_VERSION
             else:
                 raise Exception("Error getting connection to in-memory database")
         else:
-            logging.debug(f"dbfile {dbfile} exists, opening it and copying to memory")
             try:
                 conn = sqlite3.connect(dbfile)
             except Error as e:
@@ -506,6 +660,14 @@ class ExportDBInMemory(ExportDB):
             conn = sqlite3.connect(":memory:")
             conn.cursor().executescript(tempfile.read())
             conn.commit()
+            self.was_created = False
+            _, exportdb_ver = self._get_database_version(conn)
+            if exportdb_ver < OSXPHOTOS_EXPORTDB_VERSION:
+                self._create_db_tables(conn)
+                self.was_upgraded = (exportdb_ver, OSXPHOTOS_EXPORTDB_VERSION)
+            else:
+                self.was_upgraded = ()
+            self.version = OSXPHOTOS_EXPORTDB_VERSION
 
         return conn
 

osxphotos/fileutil.py

@@ -8,6 +8,7 @@ import subprocess
 import sys
 from abc import ABC, abstractmethod
 
+from .imageconverter import ImageConverter
 
 class FileUtilABC(ABC):
     """ Abstract base class for FileUtil """
@@ -29,7 +30,17 @@ class FileUtilABC(ABC):
 
     @classmethod
     @abstractmethod
-    def cmp_sig(cls, file1, file2):
+    def utime(cls, path, times):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def cmp(cls, file1, file2, mtime1=None):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def cmp_file_sig(cls, file1, file2):
         pass
 
     @classmethod
@@ -37,6 +48,11 @@ class FileUtilABC(ABC):
     def file_sig(cls, file1):
         pass
 
+    @classmethod
+    @abstractmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        pass
+
 
 class FileUtilMacOS(FileUtilABC):
     """ Various file utilities """
@@ -104,11 +120,37 @@ class FileUtilMacOS(FileUtilABC):
             os.unlink(filepath)
 
     @classmethod
-    def cmp_sig(cls, f1, s2):
+    def utime(cls, path, times):
+        """ Set the access and modified time of path. """
+        os.utime(path, times)
+
+    @classmethod
+    def cmp(cls, f1, f2, mtime1=None):
+        """Does shallow compare (file signatures) of f1 to file f2.
+        Arguments:
+        f1 --  File name
+        f2 -- File name
+        mtime1 -- optional, pass alternate file modification timestamp for f1; will be converted to int
+
+        Return value:
+        True if the file signatures as returned by stat are the same, False otherwise. 
+        Does not do a byte-by-byte comparison.
+        """
+
+        s1 = cls._sig(os.stat(f1))
+        if mtime1 is not None:
+            s1 = (s1[0], s1[1], int(mtime1))
+        s2 = cls._sig(os.stat(f2))
+        if s1[0] != stat.S_IFREG or s2[0] != stat.S_IFREG:
+            return False
+        return s1 == s2
+
+    @classmethod
+    def cmp_file_sig(cls, f1, s2):
         """Compare file f1 to signature s2.
         Arguments:
         f1 --  File name
-        s2 -- stats as returned by sig
+        s2 -- stats as returned by _sig
 
         Return value:
         True if the files are the same, False otherwise.
@@ -127,11 +169,30 @@ class FileUtilMacOS(FileUtilABC):
     def file_sig(cls, f1):
         """ return os.stat signature for file f1 """
         return cls._sig(os.stat(f1))
+    
+    @classmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        """ converts image file src_file to jpeg format as dest_file
+
+            Args:
+                src_file: image file to convert
+                dest_file: destination path to write converted file to
+                compression quality: JPEG compression quality in range 0.0 <= compression_quality <= 1.0; default 1.0 (best quality)
+                
+            Returns:
+                True if success, otherwise False 
+        """
+        converter = ImageConverter()
+        return converter.write_jpeg(src_file, dest_file, compression_quality=compression_quality)
 
     @staticmethod
     def _sig(st):
-        return (stat.S_IFMT(st.st_mode), st.st_size, st.st_mtime)
-
+        """ return tuple of (mode, size, mtime) of file based on os.stat
+            Args:
+                st: os.stat signature
+        """
+        # use int(st.st_mtime) because ditto does not copy fractional portion of mtime
+        return (stat.S_IFMT(st.st_mode), st.st_size, int(st.st_mtime))
 
 class FileUtil(FileUtilMacOS):
     """ Various file utilities """
@@ -141,8 +202,8 @@ class FileUtil(FileUtilMacOS):
 
 class FileUtilNoOp(FileUtil):
     """ No-Op implementation of FileUtil for testing / dry-run mode
-        all methods with exception of cmp_sig and file_cmp are no-op
-        cmp_sig functions as FileUtil.cmp_sig does
+        all methods with exception of cmp, cmp_file_sig and file_cmp are no-op
+        cmp and cmp_file_sig functions as FileUtil methods do
         file_cmp returns mock data
     """
 
@@ -172,7 +233,15 @@ class FileUtilNoOp(FileUtil):
     def unlink(cls, dest):
         cls.verbose(f"unlink: {dest}")
 
+    @classmethod
+    def utime(cls, path, times):
+        cls.verbose(f"utime: {path}, {times}")
+
     @classmethod
     def file_sig(cls, file1):
         cls.verbose(f"file_sig: {file1}")
         return (42, 42, 42)
+
+    @classmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        cls.verbose(f"convert_to_jpeg: {src_file}, {dest_file}, {compression_quality}")

osxphotos/imageconverter.py

@@ -0,0 +1,112 @@
+""" ImageConverter class
+    Convert an image to JPEG using CoreImage -- 
+    for example, RAW to JPEG.  Only works if Mac equipped with GPU. """
+
+# reference: https://stackoverflow.com/questions/59330149/coreimage-ciimage-write-jpg-is-shifting-colors-macos/59334308#59334308
+
+import logging
+import pathlib
+
+import Metal
+import Quartz
+from Cocoa import NSURL
+from Foundation import NSDictionary
+
+# needed to capture system-level stderr
+from wurlitzer import pipes
+
+
+class ImageConverter:
+    """ Convert images to jpeg.  This class is a singleton
+        which will re-use the Core Image CIContext to avoid
+        creating a new context for every conversion. """
+
+    def __new__(cls, *args, **kwargs):
+        """ create new object or return instance of already created singleton """
+        if not hasattr(cls, "instance") or not cls.instance:
+            cls.instance = super().__new__(cls)
+
+        return cls.instance
+
+    def __init__(self):
+        """ return existing singleton or create a new one """
+
+        if hasattr(self, "context"):
+            return
+
+        """ initialize CIContext """
+        context_options = NSDictionary.dictionaryWithDictionary_(
+            {
+                "workingColorSpace": Quartz.CoreGraphics.kCGColorSpaceExtendedSRGB,
+                "workingFormat": Quartz.kCIFormatRGBAh,
+            }
+        )
+        mtldevice = Metal.MTLCreateSystemDefaultDevice()
+        self.context = Quartz.CIContext.contextWithMTLDevice_options_(
+            mtldevice, context_options
+        )
+
+    def write_jpeg(self, input_path, output_path, compression_quality=1.0):
+        """ convert image to jpeg and write image to output_path
+
+        Args:
+            input_path: path to input image (e.g. '/path/to/import/file.CR2') as str or pathlib.Path
+            output_path: path to exported jpeg (e.g. '/path/to/export/file.jpeg') as str or pathlib.Path
+            compression_quality: JPEG compression quality, float in range 0.0 to 1.0; default is 1.0 (best quality)
+
+        Return:
+            True if conversion successful, else False
+
+        Raises:
+            ValueError if compression quality not in range 0.0 to 1.0
+            FileNotFoundError if input_path doesn't exist
+        """
+
+        # accept input_path or output_path as pathlib.Path
+        if not isinstance(input_path, str):
+            input_path = str(input_path)
+
+        if not isinstance(output_path, str):
+            output_path = str(output_path)
+
+        if not pathlib.Path(input_path).is_file():
+            raise FileNotFoundError(f"could not find {input_path}")
+
+        if not (0.0 <= compression_quality <= 1.0):
+            raise ValueError(
+                "illegal value for compression_quality: {compression_quality}"
+            )
+
+        input_url = NSURL.fileURLWithPath_(input_path)
+        output_url = NSURL.fileURLWithPath_(output_path)
+
+        with pipes() as (out, err):
+            # capture stdout and stderr from system calls
+            # otherwise, Quartz.CIImage.imageWithContentsOfURL_
+            # prints to stderr something like:
+            # 2020-09-20 20:55:25.538 python[73042:5650492] Creating client/daemon connection: B8FE995E-3F27-47F4-9FA8-559C615FD774
+            # 2020-09-20 20:55:25.652 python[73042:5650492] Got the query meta data reply for: com.apple.MobileAsset.RawCamera.Camera, response: 0
+            input_image = Quartz.CIImage.imageWithContentsOfURL_(input_url)
+
+        if input_image is None:
+            logging.debug(f"Could not create CIImage for {input_path}")
+            return False
+
+        output_colorspace = input_image.colorSpace() or Quartz.CGColorSpaceCreateWithName(
+            Quartz.CoreGraphics.kCGColorSpaceSRGB
+        )
+
+        output_options = NSDictionary.dictionaryWithDictionary_(
+            {"kCGImageDestinationLossyCompressionQuality": compression_quality}
+        )
+        _, error = self.context.writeJPEGRepresentationOfImage_toURL_colorSpace_options_error_(
+            input_image, output_url, output_colorspace, output_options, None
+        )
+        if not error:
+            return True
+        else:
+            logging.debug(
+                "Error converting file {input_path} to jpeg at {output_path}: {error}"
+            )
+            return False
+

osxphotos/photoinfo/_photoinfo_export.py

@@ -11,7 +11,6 @@
 
 # TODO: should this be its own PhotoExporter class?
 
-import filecmp
 import glob
 import json
 import logging
@@ -31,13 +30,14 @@ from .._constants import (
     _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
-from .._export_db import ExportDBNoOp
+from ..export_db import ExportDBNoOp
 from ..exiftool import ExifTool
 from ..fileutil import FileUtil
 from ..utils import dd_to_dms_str, findfiles
 
 ExportResults = namedtuple(
-    "ExportResults", ["exported", "new", "updated", "skipped", "exif_updated"]
+    "ExportResults",
+    ["exported", "new", "updated", "skipped", "exif_updated", "touched"],
 )
 
 
@@ -106,7 +106,7 @@ def _export_photo_uuid_applescript(
     )
 
     dest = pathlib.Path(dest)
-    if not dest.is_dir:
+    if not dest.is_dir():
         raise ValueError(f"dest {dest} must be a directory")
 
     if not original ^ edited:
@@ -305,6 +305,9 @@ def export2(
     export_db=None,
     fileutil=FileUtil,
     dry_run=False,
+    touch_file=False,
+    convert_to_jpeg=False,
+    jpeg_quality=1.0,
 ):
     """ export photo, like export but with update and dry_run options
         dest: must be valid destination path or exception raised 
@@ -312,10 +315,8 @@ def export2(
                     **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
                     For example, if photo is .CR2 file, edited image may be .jpeg.  
                     If you provide an extension different than what the actual file is, 
-                    export will print a warning but will export the photo using the 
-                    incorrect file extension (unless use_photos_export is true, in which case export will
-                    use the extension provided by Photos upon export; in this case, an incorrect extension is
-                    silently ignored).
+                    will export the photo using the incorrect file extension (unless use_photos_export is true, 
+                    in which case export will use the extension provided by Photos upon export.
                     e.g. to get the extension of the edited photo, 
                     reference PhotoInfo.path_edited
         edited: (boolean, default=False); if True will export the edited version of the photo 
@@ -334,7 +335,6 @@ def export2(
         timeout: (int, default=120) timeout in seconds used with use_photos_export
         exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
         no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
-        returns list of full paths to the exported files
         use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
         when exporting metadata with exiftool or sidecar
         use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
@@ -347,6 +347,9 @@ def export2(
                 for getting/setting data related to exported files to compare update state
         fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
         dry_run: (boolean, default=False); set to True to run in "dry run" mode
+        touch_file: (boolean, default=False); if True, sets file's modification time upon photo date
+        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
         Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
                     where each field is a list of file paths
@@ -355,6 +358,10 @@ def export2(
               and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp)
             """
 
+    # NOTE: This function is very complex and does a lot of things.
+    # Don't modify this code if you don't fully understand everything it does.
+    # TODO: This is a good candidate for refactoring.
+
     # when called from export(), won't get an export_db, so use no-op version
     if export_db is None:
         export_db = ExportDBNoOp()
@@ -375,6 +382,9 @@ def export2(
     # list of all files skipped because they do not need to be updated (for use with update=True)
     update_skipped_files = []
 
+    # list of all files with utime touched (touch_file = True)
+    touched_files = []
+
     # check edited and raise exception trying to export edited version of
     # photo that hasn't been edited
     if edited and not self.hasadjustments:
@@ -387,34 +397,41 @@ def export2(
         raise TypeError(
             "Too many positional arguments.  Should be at most two: destination, filename."
         )
-    else:
-        # verify destination is a valid path
-        if dest is None:
-            raise ValueError("Destination must not be None")
-        elif not dry_run and not os.path.isdir(dest):
-            raise FileNotFoundError("Invalid path passed to export")
 
-        if filename and len(filename) == 1:
-            # if filename passed, use it
-            fname = filename[0]
-        else:
-            # no filename provided so use the default
-            # if edited file requested, use filename but add _edited
-            # need to use file extension from edited file as Photos saves a jpeg once edited
-            if edited and not use_photos_export:
-                # verify we have a valid path_edited and use that to get filename
-                if not self.path_edited:
-                    raise FileNotFoundError(
-                        "edited=True but path_edited is none; hasadjustments: "
-                        f" {self.hasadjustments}"
-                    )
-                edited_name = pathlib.Path(self.path_edited).name
-                edited_suffix = pathlib.Path(edited_name).suffix
-                fname = (
-                    pathlib.Path(self.filename).stem + edited_identifier + edited_suffix
+    # verify destination is a valid path
+    if dest is None:
+        raise ValueError("Destination must not be None")
+    elif not dry_run and not os.path.isdir(dest):
+        raise FileNotFoundError("Invalid path passed to export")
+
+    if filename and len(filename) == 1:
+        # if filename passed, use it
+        fname = filename[0]
+    else:
+        # no filename provided so use the default
+        # if edited file requested, use filename but add _edited
+        # need to use file extension from edited file as Photos saves a jpeg once edited
+        if edited and not use_photos_export:
+            # verify we have a valid path_edited and use that to get filename
+            if not self.path_edited:
+                raise FileNotFoundError(
+                    "edited=True but path_edited is none; hasadjustments: "
+                    f" {self.hasadjustments}"
                 )
-            else:
-                fname = self.filename
+            edited_name = pathlib.Path(self.path_edited).name
+            edited_suffix = pathlib.Path(edited_name).suffix
+            fname = pathlib.Path(self.filename).stem + edited_identifier + edited_suffix
+        else:
+            fname = self.filename
+
+    uti = self.uti if edited else self.uti_original
+    if convert_to_jpeg and self.isphoto and uti != "public.jpeg":
+        # not a jpeg but will convert to jpeg upon export so fix file extension
+        fname_new = pathlib.Path(fname)
+        fname = str(fname_new.parent / f"{fname_new.stem}.jpeg")
+    else:
+        # nothing to convert
+        convert_to_jpeg = False
 
     # check destination path
     dest = pathlib.Path(dest)
@@ -468,21 +485,17 @@ def export2(
             raise FileNotFoundError(f"{src} does not appear to exist")
 
         if not _check_export_suffix(src, dest, edited):
-            logging.warning(
+            logging.debug(
                 f"Invalid destination suffix: {dest.suffix} for {self.path}, "
                 + f"edited={edited}, path_edited={self.path_edited}, "
                 + f"original_filename={self.original_filename}, filename={self.filename}"
             )
 
-        logging.debug(
-            f"exporting {src} to {dest}, overwrite={overwrite}, increment={increment}, dest exists: {dest.exists()}"
-        )
-
         # found source now try to find right destination
         if update and dest.exists():
             # destination exists, check to see if destination is the right UUID
             dest_uuid = export_db.get_uuid_for_file(dest)
-            if dest_uuid is None and filecmp.cmp(src, dest):
+            if dest_uuid is None and fileutil.cmp(src, dest):
                 # might be exporting into a pre-ExportDB folder or the DB got deleted
                 logging.debug(
                     f"Found matching file with blank uuid: {self.uuid}, {dest}"
@@ -493,14 +506,13 @@ def export2(
                     self.uuid,
                     fileutil.file_sig(dest),
                     (None, None, None),
+                    (None, None, None),
+                    (None, None, None),
                     self.json(),
                     None,
                 )
             if dest_uuid != self.uuid:
                 # not the right file, find the right one
-                logging.debug(
-                    f"Need to find right photo: uuid={self.uuid}, dest={dest_uuid}, dest={dest}, path={self.path}"
-                )
                 count = 1
                 glob_str = str(dest.parent / f"{dest.stem} (*{dest.suffix}")
                 dest_files = glob.glob(glob_str)
@@ -508,17 +520,11 @@ def export2(
                 for file_ in dest_files:
                     dest_uuid = export_db.get_uuid_for_file(file_)
                     if dest_uuid == self.uuid:
-                        logging.debug(
-                            f"Found matching file for uuid: {dest_uuid}, {file_}"
-                        )
                         dest = pathlib.Path(file_)
                         found_match = True
                         break
-                    elif dest_uuid is None and filecmp.cmp(src, file_):
+                    elif dest_uuid is None and fileutil.cmp(src, file_):
                         # files match, update the UUID
-                        logging.debug(
-                            f"Found matching file with blank uuid: {self.uuid}, {file_}"
-                        )
                         dest = pathlib.Path(file_)
                         found_match = True
                         export_db.set_data(
@@ -526,16 +532,14 @@ def export2(
                             self.uuid,
                             fileutil.file_sig(dest),
                             (None, None, None),
+                            (None, None, None),
+                            (None, None, None),
                             self.json(),
                             None,
                         )
                         break
 
                 if not found_match:
-                    logging.debug(
-                        f"Didn't find destination match for uuid {self.uuid} {dest}"
-                    )
-
                     # increment the destination file
                     count = 1
                     glob_str = str(dest.parent / f"{dest.stem}*")
@@ -546,7 +550,6 @@ def export2(
                         dest_new = f"{dest.stem} ({count})"
                         count += 1
                     dest = dest.parent / f"{dest_new}{dest.suffix}"
-                    logging.debug(f"New destination = {dest}, uuid = {self.uuid}")
 
         # export the dest file
         results = self._export_photo(
@@ -558,12 +561,17 @@ def export2(
             no_xattr,
             export_as_hardlink,
             exiftool,
-            fileutil,
+            touch_file,
+            convert_to_jpeg,
+            fileutil=fileutil,
+            edited=edited,
+            jpeg_quality=jpeg_quality,
         )
         exported_files = results.exported
         update_new_files = results.new
         update_updated_files = results.updated
         update_skipped_files = results.skipped
+        touched_files = results.touched
 
         # copy live photo associated .mov if requested
         if live_photo and self.live_photo:
@@ -583,12 +591,16 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
-                    fileutil,
+                    touch_file,
+                    convert_to_jpeg,
+                    fileutil=fileutil,
+                    jpeg_quality=jpeg_quality,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
+                touched_files.extend(results.touched)
             else:
                 logging.debug(f"Skipping missing live movie for {filename}")
 
@@ -608,17 +620,21 @@ def export2(
                     no_xattr,
                     export_as_hardlink,
                     exiftool,
-                    fileutil,
+                    touch_file,
+                    convert_to_jpeg,
+                    fileutil=fileutil,
+                    jpeg_quality=jpeg_quality,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
                 update_updated_files.extend(results.updated)
                 update_skipped_files.extend(results.skipped)
+                touched_files.extend(results.touched)
             else:
                 logging.debug(f"Skipping missing RAW photo for {filename}")
     else:
         # use_photo_export
-        exported = None
+        exported = []
         # export live_photo .mov file?
         live_photo = True if live_photo and self.live_photo else False
         if edited:
@@ -628,7 +644,7 @@ def export2(
                 filestem = dest.stem
             else:
                 # didn't get passed a filename, add _edited
-                filestem = f"{dest.stem}_edited"
+                filestem = f"{dest.stem}{edited_identifier}"
                 dest = dest.parent / f"{filestem}.jpeg"
 
             exported = _export_photo_uuid_applescript(
@@ -657,19 +673,25 @@ def export2(
                 dry_run=dry_run,
             )
 
-        if exported is not None:
+        if exported:
+            if touch_file:
+                for exported_file in exported:
+                    touched_files.append(exported_file)
+                    ts = int(self.date.timestamp())
+                    fileutil.utime(exported_file, (ts, ts))
             exported_files.extend(exported)
+            if update:
+                update_new_files.extend(exported)
+
         else:
             logging.warning(
                 f"Error exporting photo {self.uuid} to {dest} with use_photos_export"
             )
 
     # export metadata
-    info = export_db.get_info_for_uuid(self.uuid)
-
     if sidecar_json:
         logging.debug("writing exiftool_json_sidecar")
-        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.json")
+        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.json")
         sidecar_str = self._exiftool_json_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
             use_persons_as_keywords=use_persons_as_keywords,
@@ -685,12 +707,13 @@ def export2(
 
     if sidecar_xmp:
         logging.debug("writing xmp_sidecar")
-        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}.xmp")
+        sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.xmp")
         sidecar_str = self._xmp_sidecar(
             use_albums_as_keywords=use_albums_as_keywords,
             use_persons_as_keywords=use_persons_as_keywords,
             keyword_template=keyword_template,
             description_template=description_template,
+            extension=dest.suffix[1:] if dest.suffix else None,
         )
         if not dry_run:
             try:
@@ -727,7 +750,6 @@ def export2(
             if old_data is None or files_are_different:
                 # didn't have old data, assume we need to write it
                 # or files were different
-                logging.debug(f"No exifdata for {exported_file}, writing it")
                 if not dry_run:
                     self._write_exif_data(
                         exported_file,
@@ -751,7 +773,6 @@ def export2(
                 exif_files_updated.append(exported_file)
     elif exiftool and exif_files:
         for exported_file in exif_files:
-            logging.debug(f"Writing exif data to {exported_file}")
             if not dry_run:
                 self._write_exif_data(
                     exported_file,
@@ -760,6 +781,7 @@ def export2(
                     keyword_template=keyword_template,
                     description_template=description_template,
                 )
+
             export_db.set_exifdata_for_file(
                 exported_file,
                 self._exiftool_json_sidecar(
@@ -774,13 +796,23 @@ def export2(
             )
             exif_files_updated.append(exported_file)
 
-    return ExportResults(
+    if touch_file:
+        for exif_file in exif_files_updated:
+            touched_files.append(exif_file)
+            ts = int(self.date.timestamp())
+            fileutil.utime(exif_file, (ts, ts))
+
+    touched_files = list(set(touched_files))
+
+    results = ExportResults(
         exported_files,
         update_new_files,
         update_updated_files,
         update_skipped_files,
         exif_files_updated,
+        touched_files,
     )
+    return results
 
 
 def _export_photo(
@@ -793,11 +825,15 @@ def _export_photo(
     no_xattr,
     export_as_hardlink,
     exiftool,
+    touch_file,
+    convert_to_jpeg,
     fileutil=FileUtil,
+    edited=False,
+    jpeg_quality=1.0,
 ):
     """ Helper function for export()
         Does the actual copy or hardlink taking the appropriate 
-        action depending on update, overwrite
+        action depending on update, overwrite, export_as_hardlink
         Assumes destination is the right destination (e.g. UUID matches)
         sets UUID and JSON info foo exported file using set_uuid_for_file, set_inf_for_uuido
     
@@ -810,153 +846,146 @@ def _export_photo(
         no_xattr: don't copy extended attributes
         export_as_hardlink: bool
         exiftool: bool
+        touch_file: bool
+        convert_to_jpeg: bool; if True, convert file to jpeg on export
         fileutil: FileUtil class that conforms to fileutil.FileUtilABC
+        edited: bool; set to True if exporting edited version of photo
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
     Returns:
         ExportResults
+
+    Raises:
+        ValueError if export_as_hardlink and convert_to_jpeg both True
     """
 
+    if export_as_hardlink and convert_to_jpeg:
+        raise ValueError("export_as_hardlink and convert_to_jpeg cannot both be True")
+
     exported_files = []
     update_updated_files = []
     update_new_files = []
     update_skipped_files = []
+    touched_files = []
 
     dest_str = str(dest)
     dest_exists = dest.exists()
-    if export_as_hardlink:
-        # use hardlink instead of copy
-        if not update:
-            # not update, do the the hardlink
-            if overwrite and dest.exists():
-                # need to remove the destination first
-                # dest.unlink()
-                fileutil.unlink(dest)
-            logging.debug(f"Not update: export_as_hardlink linking file {src} {dest}")
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-        elif dest_exists and dest.samefile(src):
-            # update, hardlink and it already points to the right file, do nothing
-            logging.debug(
-                f"Update: skipping samefile with export_as_hardlink {src} {dest}"
-            )
-            update_skipped_files.append(dest_str)
-        elif dest_exists:
-            # update, not the same file (e.g. user may not have used export_as_hardlink last time it was run
-            logging.debug(
-                f"Update: removing existing file prior to export_as_hardlink {src} {dest}"
-            )
-            # dest.unlink()
-            fileutil.unlink(dest)
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            update_updated_files.append(dest_str)
-            exported_files.append(dest_str)
+    op_desc = "export_as_hardlink" if export_as_hardlink else "export_by_copying"
+
+    if update:  # updating
+        cmp_touch, cmp_orig = False, False
+        if dest_exists:
+            # update, destination exists, but we might not need to replace it...
+            if exiftool:
+                sig_exif = export_db.get_stat_exif_for_file(dest_str)
+                cmp_orig = fileutil.cmp_file_sig(dest_str, sig_exif)
+                sig_exif = (sig_exif[0], sig_exif[1], int(self.date.timestamp()))
+                cmp_touch = fileutil.cmp_file_sig(dest_str, sig_exif)
+            elif convert_to_jpeg:
+                sig_converted = export_db.get_stat_converted_for_file(dest_str)
+                cmp_orig = fileutil.cmp_file_sig(dest_str, sig_converted)
+                sig_converted = (
+                    sig_converted[0],
+                    sig_converted[1],
+                    int(self.date.timestamp()),
+                )
+                cmp_touch = fileutil.cmp_file_sig(dest_str, sig_converted)
+            else:
+                cmp_orig = fileutil.cmp(src, dest)
+                cmp_touch = fileutil.cmp(src, dest, mtime1=int(self.date.timestamp()))
+
+            sig_cmp = cmp_touch if touch_file else cmp_orig
+
+            if edited:
+                # requested edited version of photo
+                # need to see if edited version in Photos library has changed
+                # (e.g. it's been edited again)
+                sig_edited = export_db.get_stat_edited_for_file(dest_str)
+                cmp_edited = (
+                    fileutil.cmp_file_sig(src, sig_edited)
+                    if sig_edited != (None, None, None)
+                    else False
+                )
+                sig_cmp = sig_cmp and cmp_edited
+
+            if (export_as_hardlink and dest.samefile(src)) or (
+                not export_as_hardlink and not dest.samefile(src) and sig_cmp
+            ):
+                # destination exists and signatures match, skip it
+                update_skipped_files.append(dest_str)
+            else:
+                # destination exists but signature is different
+                if touch_file and cmp_orig and not cmp_touch:
+                    # destination exists, signature matches original but does not match expected touch time
+                    # skip exporting but update touch time
+                    update_skipped_files.append(dest_str)
+                    touched_files.append(dest_str)
+                elif not touch_file and cmp_touch and not cmp_orig:
+                    # destination exists, signature matches expected touch but not original
+                    # user likely exported with touch_file and is now exporting without touch_file
+                    # don't update the file because it's same but leave touch time
+                    update_skipped_files.append(dest_str)
+                else:
+                    # destination exists but is different
+                    update_updated_files.append(dest_str)
+                    if touch_file:
+                        touched_files.append(dest_str)
+
         else:
-            # update, hardlink, destination doesn't exist (new file)
-            logging.debug(
-                f"Update: exporting new file with export_as_hardlink {src} {dest}"
-            )
-            fileutil.hardlink(src, dest)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
+            # update, destination doesn't exist (new file)
+            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
             update_new_files.append(dest_str)
+            if touch_file:
+                touched_files.append(dest_str)
     else:
-        if not update:
-            # not update, do the the copy
-            if overwrite and dest.exists():
-                # need to remove the destination first
-                # dest.unlink()
-                fileutil.unlink(dest)
-            logging.debug(f"Not update: copying file {src} {dest}")
-            fileutil.copy(src, dest_str, norsrc=no_xattr)
-            exported_files.append(dest_str)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
+        # not update, export the file
+        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
+        exported_files.append(dest_str)
+        if touch_file:
+            sig = fileutil.file_sig(src)
+            sig = (sig[0], sig[1], int(self.date.timestamp()))
+            if not fileutil.cmp_file_sig(src, sig):
+                touched_files.append(dest_str)
+    if not update_skipped_files:
+        converted_stat = (None, None, None)
+        edited_stat = fileutil.file_sig(src) if edited else (None, None, None)
+        if dest_exists and (update or overwrite):
+            # need to remove the destination first
+            logging.debug(
+                f"Update: removing existing file prior to {op_desc} {src} {dest}"
             )
-        # elif dest_exists and not exiftool and cmp_file(dest_str, export_db.get_stat_orig_for_file(dest_str)):
-        elif (
-            dest_exists
-            and not exiftool
-            and filecmp.cmp(src, dest)
-            and not dest.samefile(src)
-        ):
-            # destination exists but is identical
-            logging.debug(f"Update: skipping identifical original files {src} {dest}")
-            # call set_stat because code can reach this spot if no export DB but exporting a RAW or live photo
-            # potentially re-writes the data in the database but ensures database is complete
-            export_db.set_stat_orig_for_file(dest_str, fileutil.file_sig(dest_str))
-            update_skipped_files.append(dest_str)
-        elif (
-            dest_exists
-            and exiftool
-            and fileutil.cmp_sig(dest_str, export_db.get_stat_exif_for_file(dest_str))
-            and not dest.samefile(src)
-        ):
-            # destination exists but is identical
-            logging.debug(f"Update: skipping identifical exiftool files {src} {dest}")
-            update_skipped_files.append(dest_str)
-        elif dest_exists:
-            # destination exists but is different or is a hardlink
-            logging.debug(f"Update: removing existing file prior to copy {src} {dest}")
-            stat_src = os.stat(src)
-            stat_dest = os.stat(dest)
-            # dest.unlink()
             fileutil.unlink(dest)
-            fileutil.copy(src, dest_str, norsrc=no_xattr)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-            update_updated_files.append(dest_str)
+        if export_as_hardlink:
+            fileutil.hardlink(src, dest)
+        elif convert_to_jpeg:
+            # use convert_to_jpeg to export the file
+            fileutil.convert_to_jpeg(src, dest_str, compression_quality=jpeg_quality)
+            converted_stat = fileutil.file_sig(dest_str)
         else:
-            # destination doesn't exist, copy the file
-            logging.debug(f"Update: copying new file {src} {dest}")
             fileutil.copy(src, dest_str, norsrc=no_xattr)
-            export_db.set_data(
-                dest_str,
-                self.uuid,
-                fileutil.file_sig(dest_str),
-                (None, None, None),
-                self.json(),
-                None,
-            )
-            exported_files.append(dest_str)
-            update_new_files.append(dest_str)
+
+        export_db.set_data(
+            dest_str,
+            self.uuid,
+            fileutil.file_sig(dest_str),
+            (None, None, None),
+            converted_stat,
+            edited_stat,
+            self.json(),
+            None,
+        )
+
+    if touched_files:
+        ts = int(self.date.timestamp())
+        fileutil.utime(dest, (ts, ts))
 
     return ExportResults(
-        exported_files, update_new_files, update_updated_files, update_skipped_files, []
+        exported_files + update_new_files + update_updated_files,
+        update_new_files,
+        update_updated_files,
+        update_skipped_files,
+        [],
+        touched_files,
     )
 
 
@@ -1132,6 +1161,7 @@ def _xmp_sidecar(
     use_persons_as_keywords=False,
     keyword_template=None,
     description_template=None,
+    extension=None,
 ):
     """ returns string for XMP sidecar 
         use_albums_as_keywords: treat album names as keywords
@@ -1139,10 +1169,12 @@ def _xmp_sidecar(
         keyword_template: (list of strings); list of template strings to render as keywords 
         description_template: string; optional template string that will be rendered for use as photo description """
 
-    # TODO: add additional fields to XMP file?
-
     xmp_template = Template(filename=os.path.join(_TEMPLATE_DIR, _XMP_TEMPLATE_NAME))
 
+    if extension is None:
+        extension = pathlib.Path(self.original_filename)
+        extension = extension.suffix[1:] if extension.suffix else None
+
     if description_template is not None:
         description = self.render_template(
             description_template, expand_inplace=True, inplace_sep=", "
@@ -1211,6 +1243,7 @@ def _xmp_sidecar(
         keywords=keyword_list,
         persons=person_list,
         subjects=subject_list,
+        extension=extension,
     )
 
     # remove extra lines that mako inserts from template

osxphotos/photoinfo/photoinfo.py

@@ -5,16 +5,12 @@ PhotosDB.photos() returns a list of PhotoInfo objects
 """
 
 import dataclasses
-import glob
 import json
 import logging
 import os
 import os.path
 import pathlib
-import subprocess
-import sys
 from datetime import timedelta, timezone
-from pprint import pformat
 
 import yaml
 
@@ -24,11 +20,13 @@ from .._constants import (
     _PHOTOS_4_ALBUM_KIND,
     _PHOTOS_4_ROOT_FOLDER,
     _PHOTOS_4_VERSION,
+    _PHOTOS_5_VERSION,
     _PHOTOS_5_ALBUM_KIND,
+    _PHOTOS_5_IMPORT_SESSION_ALBUM_KIND,
     _PHOTOS_5_SHARED_ALBUM_KIND,
     _PHOTOS_5_SHARED_PHOTO_PATH,
 )
-from ..albuminfo import AlbumInfo
+from ..albuminfo import AlbumInfo, ImportInfo
 from ..personinfo import FaceInfo, PersonInfo
 from ..phototemplate import PhotoTemplate
 from ..placeinfo import PlaceInfo4, PlaceInfo5
@@ -70,11 +68,9 @@ class PhotoInfo:
     @property
     def filename(self):
         """ filename of the picture """
-        # sourcery off
-        if self.has_raw and self.raw_original:
-            # return name of the RAW file
-            # TODO: not yet implemented
-            return self._info["filename"]
+        if self._db._db_version <= _PHOTOS_4_VERSION and self.has_raw and self.raw_original:
+            # return the JPEG version as that's what Photos 5+ does
+            return self._info["raw_pair_info"]["filename"]
         else:
             return self._info["filename"]
 
@@ -82,13 +78,17 @@ class PhotoInfo:
     def original_filename(self):
         """ original filename of the picture 
             Photos 5 mangles filenames upon import """
-        return self._info["originalFilename"]
+        if self._db._db_version <= _PHOTOS_4_VERSION and self.has_raw and self.raw_original:
+            # return the JPEG version as that's what Photos 5+ does
+            return self._info["raw_pair_info"]["originalFilename"]
+        else:
+            return self._info["originalFilename"]
 
     @property
     def date(self):
         """ image creation date as timezone aware datetime object """
         return self._info["imageDate"]
-        
+
     @property
     def date_modified(self):
         """ image modification date as timezone aware datetime object
@@ -110,39 +110,67 @@ class PhotoInfo:
     @property
     def path(self):
         """ absolute path on disk of the original picture """
+        try:
+            return self._path
+        except AttributeError:
+            self._path = None
+            photopath = None
+            if self._info["isMissing"] == 1:
+                return photopath  # path would be meaningless until downloaded
 
-        photopath = None
-        if self._info["isMissing"] == 1:
-            return photopath  # path would be meaningless until downloaded
+            if self._db._db_version <= _PHOTOS_4_VERSION:
+                if self._info["has_raw"]:
+                    # return the path to JPEG even if RAW is original
+                    vol = (
+                        self._db._dbvolumes[self._info["raw_pair_info"]["volumeId"]]
+                        if self._info["raw_pair_info"]["volumeId"] is not None
+                        else None
+                    )
+                    if vol is not None:
+                        photopath = os.path.join(
+                            "/Volumes", vol, self._info["raw_pair_info"]["imagePath"]
+                        )
+                    else:
+                        photopath = os.path.join(
+                            self._db._masters_path,
+                            self._info["raw_pair_info"]["imagePath"],
+                        )
+                else:
+                    vol = self._info["volume"]
+                    if vol is not None:
+                        photopath = os.path.join(
+                            "/Volumes", vol, self._info["imagePath"]
+                        )
+                    else:
+                        photopath = os.path.join(
+                            self._db._masters_path, self._info["imagePath"]
+                        )
+                self._path = photopath
+                return photopath
 
-        if self._db._db_version <= _PHOTOS_4_VERSION:
-            vol = self._info["volume"]
-            if vol is not None:
-                photopath = os.path.join("/Volumes", vol, self._info["imagePath"])
+            if self._info["shared"]:
+                # shared photo
+                photopath = os.path.join(
+                    self._db._library_path,
+                    _PHOTOS_5_SHARED_PHOTO_PATH,
+                    self._info["directory"],
+                    self._info["filename"],
+                )
+                self._path = photopath
+                return photopath
+
+            if self._info["directory"].startswith("/"):
+                photopath = os.path.join(
+                    self._info["directory"], self._info["filename"]
+                )
             else:
                 photopath = os.path.join(
-                    self._db._masters_path, self._info["imagePath"]
+                    self._db._masters_path,
+                    self._info["directory"],
+                    self._info["filename"],
                 )
+            self._path = photopath
             return photopath
-            # TODO: Is there a way to use applescript or PhotoKit to force the download in this
-
-        if self._info["shared"]:
-            # shared photo
-            photopath = os.path.join(
-                self._db._library_path,
-                _PHOTOS_5_SHARED_PHOTO_PATH,
-                self._info["directory"],
-                self._info["filename"],
-            )
-            return photopath
-
-        if self._info["directory"].startswith("/"):
-            photopath = os.path.join(self._info["directory"], self._info["filename"])
-        else:
-            photopath = os.path.join(
-                self._db._masters_path, self._info["directory"], self._info["filename"]
-            )
-        return photopath
 
     @property
     def path_edited(self):
@@ -152,109 +180,132 @@ class PhotoInfo:
         # TODO: break this code into a _path_edited_4 and _path_edited_5
         # version to simplify the big if/then; same for path_live_photo
 
-        photopath = None
-
-        if self._db._db_version <= _PHOTOS_4_VERSION:
-            if self._info["hasAdjustments"]:
-                edit_id = self._info["edit_resource_id"]
-                if edit_id is not None:
-                    library = self._db._library_path
-                    folder_id, file_id = _get_resource_loc(edit_id)
-                    # todo: is this always true or do we need to search file file_id under folder_id
-                    # figure out what kind it is and build filename
-                    filename = None
-                    if self._info["type"] == _PHOTO_TYPE:
-                        # it's a photo
-                        filename = f"fullsizeoutput_{file_id}.jpeg"
-                    elif self._info["type"] == _MOVIE_TYPE:
-                        # it's a movie
-                        filename = f"fullsizeoutput_{file_id}.mov"
-                    else:
-                        # don't know what it is!
-                        logging.debug(f"WARNING: unknown type {self._info['type']}")
-                        return None
-
-                    # photopath appears to usually be in "00" subfolder but
-                    # could be elsewhere--I haven't figured out this logic yet
-                    # first see if it's in 00
-                    photopath = os.path.join(
-                        library,
-                        "resources",
-                        "media",
-                        "version",
-                        folder_id,
-                        "00",
-                        filename,
-                    )
-
-                    if not os.path.isfile(photopath):
-                        rootdir = os.path.join(
-                            library, "resources", "media", "version", folder_id
-                        )
-
-                        for dirname, _, filelist in os.walk(rootdir):
-                            if filename in filelist:
-                                photopath = os.path.join(dirname, filename)
-                                break
-
-                    # check again to see if we found a valid file
-                    if not os.path.isfile(photopath):
-                        logging.debug(
-                            f"MISSING PATH: edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
-                        )
-                        photopath = None
-                else:
-                    logging.debug(
-                        f"{self.uuid} hasAdjustments but edit_resource_id is None"
-                    )
-                    photopath = None
+        try:
+            return self._path_edited
+        except AttributeError:
+            if self._db._db_version <= _PHOTOS_4_VERSION:
+                self._path_edited = self._path_edited_4()
+                return self._path_edited
             else:
+                self._path_edited = self._path_edited_5()
+                return self._path_edited
+
+    def _path_edited_5(self):
+        """ return path_edited for Photos >= 5 """
+        # In Photos 5.0 / Catalina / MacOS 10.15:
+        # edited photos appear to always be converted to .jpeg and stored in
+        # library_name/resources/renders/X/UUID_1_201_a.jpeg
+        # where X = first letter of UUID
+        # and UUID = UUID of image
+        # this seems to be true even for photos not copied to Photos library and
+        # where original format was not jpg/jpeg
+        # if more than one edit, previous edit is stored as UUID_p.jpeg
+        #
+        # In Photos 6.0 / Big Sur, the edited image is a .heic if the photo isn't a jpeg,
+        # otherwise it's a jpeg.  It could also be a jpeg if photo library upgraded from earlier
+        # version.
+
+        if self._db._db_version < _PHOTOS_5_VERSION:
+            raise RuntimeError("Wrong database format!")
+
+        if self._info["hasAdjustments"]:
+            library = self._db._library_path
+            directory = self._uuid[0]  # first char of uuid
+            filename = None
+            if self._info["type"] == _PHOTO_TYPE:
+                # it's a photo
+                if self._db._photos_ver == 5:
+                    filename = f"{self._uuid}_1_201_a.jpeg"
+                else:
+                    # could be a heic or a jpeg
+                    if self.uti == "public.heic":
+                        filename = f"{self._uuid}_1_201_a.heic"
+                    else:
+                        filename = f"{self._uuid}_1_201_a.jpeg"
+            elif self._info["type"] == _MOVIE_TYPE:
+                # it's a movie
+                filename = f"{self._uuid}_2_0_a.mov"
+            else:
+                # don't know what it is!
+                logging.debug(f"WARNING: unknown type {self._info['type']}")
+                return None
+
+            photopath = os.path.join(
+                library, "resources", "renders", directory, filename
+            )
+
+            if not os.path.isfile(photopath):
+                logging.debug(
+                    f"edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
+                )
                 photopath = None
-
-            # if self._info["isMissing"] == 1:
-            #     photopath = None  # path would be meaningless until downloaded
         else:
-            # in Photos 5.0 / Catalina / MacOS 10.15:
-            # edited photos appear to always be converted to .jpeg and stored in
-            # library_name/resources/renders/X/UUID_1_201_a.jpeg
-            # where X = first letter of UUID
-            # and UUID = UUID of image
-            # this seems to be true even for photos not copied to Photos library and
-            # where original format was not jpg/jpeg
-            # if more than one edit, previous edit is stored as UUID_p.jpeg
+            photopath = None
 
-            if self._info["hasAdjustments"]:
+        # TODO: might be possible for original/master to be missing but edit to still be there
+        # if self._info["isMissing"] == 1:
+        #     photopath = None  # path would be meaningless until downloaded
+
+        # logging.debug(photopath)
+
+        return photopath
+
+    def _path_edited_4(self):
+        """ return path_edited for Photos <= 4 """
+
+        if self._db._db_version > _PHOTOS_4_VERSION:
+            raise RuntimeError("Wrong database format!")
+
+        photopath = None
+        if self._info["hasAdjustments"]:
+            edit_id = self._info["edit_resource_id"]
+            if edit_id is not None:
                 library = self._db._library_path
-                directory = self._uuid[0]  # first char of uuid
+                folder_id, file_id = _get_resource_loc(edit_id)
+                # todo: is this always true or do we need to search file file_id under folder_id
+                # figure out what kind it is and build filename
                 filename = None
                 if self._info["type"] == _PHOTO_TYPE:
                     # it's a photo
-                    filename = f"{self._uuid}_1_201_a.jpeg"
+                    filename = f"fullsizeoutput_{file_id}.jpeg"
                 elif self._info["type"] == _MOVIE_TYPE:
                     # it's a movie
-                    filename = f"{self._uuid}_2_0_a.mov"
+                    filename = f"fullsizeoutput_{file_id}.mov"
                 else:
                     # don't know what it is!
                     logging.debug(f"WARNING: unknown type {self._info['type']}")
                     return None
 
+                # photopath appears to usually be in "00" subfolder but
+                # could be elsewhere--I haven't figured out this logic yet
+                # first see if it's in 00
                 photopath = os.path.join(
-                    library, "resources", "renders", directory, filename
+                    library, "resources", "media", "version", folder_id, "00", filename,
                 )
 
+                if not os.path.isfile(photopath):
+                    rootdir = os.path.join(
+                        library, "resources", "media", "version", folder_id
+                    )
+
+                    for dirname, _, filelist in os.walk(rootdir):
+                        if filename in filelist:
+                            photopath = os.path.join(dirname, filename)
+                            break
+
+                # check again to see if we found a valid file
                 if not os.path.isfile(photopath):
                     logging.debug(
-                        f"edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
+                        f"MISSING PATH: edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
                     )
                     photopath = None
             else:
+                logging.debug(
+                    f"{self.uuid} hasAdjustments but edit_resource_id is None"
+                )
                 photopath = None
-
-            # TODO: might be possible for original/master to be missing but edit to still be there
-            # if self._info["isMissing"] == 1:
-            #     photopath = None  # path would be meaningless until downloaded
-
-            # logging.debug(photopath)
+        else:
+            photopath = None
 
         return photopath
 
@@ -357,7 +408,7 @@ class PhotoInfo:
         except AttributeError:
             try:
                 faces = self._db._db_faceinfo_uuid[self._uuid]
-                self._faceinfo =  [FaceInfo(db=self._db, pk=pk) for pk in faces]
+                self._faceinfo = [FaceInfo(db=self._db, pk=pk) for pk in faces]
             except KeyError:
                 # no faces
                 self._faceinfo = []
@@ -387,6 +438,19 @@ class PhotoInfo:
             ]
             return self._album_info
 
+    @property
+    def import_info(self):
+        """ ImportInfo object representing import session for the photo or None if no import session """
+        try:
+            return self._import_info
+        except AttributeError:
+            self._import_info = (
+                ImportInfo(db=self._db, uuid=self._info["import_uuid"])
+                if self._info["import_uuid"] is not None
+                else None
+            )
+            return self._import_info
+
     @property
     def keywords(self):
         """ list of keywords for picture """
@@ -463,7 +527,37 @@ class PhotoInfo:
         """ Returns Uniform Type Identifier (UTI) for the image
             for example: public.jpeg or com.apple.quicktime-movie
         """
-        return self._info["UTI"]
+        if self._db._db_version <= _PHOTOS_4_VERSION:
+            if self.hasadjustments:
+                return self._info["UTI_edited"]
+            elif self.has_raw and self.raw_original:
+                # return UTI of the non-raw image to match Photos 5+ behavior
+                return self._info["raw_pair_info"]["UTI"]
+            else:
+                return self._info["UTI"]
+        else:
+            return self._info["UTI"]
+
+    @property
+    def uti_original(self):
+        """ Returns Uniform Type Identifier (UTI) for the original image
+            for example: public.jpeg or com.apple.quicktime-movie
+        """
+        if self._db._db_version <= _PHOTOS_4_VERSION and self._info["has_raw"]:
+            return self._info["raw_pair_info"]["UTI"]
+        else:
+            return self._info["UTI_original"]
+
+    @property
+    def uti_edited(self):
+        """ Returns Uniform Type Identifier (UTI) for the edited image 
+            if the photo has been edited, otherwise None; 
+            for example: public.jpeg 
+        """
+        if self._db._db_version >= _PHOTOS_5_VERSION:
+            return self.uti if self.hasadjustments else None
+        else:
+            return self._info["UTI_edited"]
 
     @property
     def uti_raw(self):
@@ -654,12 +748,17 @@ class PhotoInfo:
 
     @property
     def has_raw(self):
-        """ returns True if photo has an associated RAW image, otherwise False """
+        """ returns True if photo has an associated raw image (that is, it's a RAW+JPEG pair), otherwise False """
         return self._info["has_raw"]
 
+    @property
+    def israw(self):
+        """ returns True if photo is a raw image. For images with an associated RAW+JPEG pair, see has_raw """
+        return "raw-image" in self.uti_original
+
     @property
     def raw_original(self):
-        """ returns True if associated RAW image and the RAW image is selected in Photos
+        """ returns True if associated raw image and the raw image is selected in Photos
             via "Use RAW as Original "
             otherwise returns False """
         return self._info["raw_is_original"]
@@ -745,7 +844,7 @@ class PhotoInfo:
         """ Return list of album UUIDs this photo is found in
         
             Filters out albums in the trash and any special album types
-        
+
         Returns: list of album UUIDs 
         """
         if self._db._db_version <= _PHOTOS_4_VERSION:

osxphotos/photosdb/_photosdb_process_faceinfo.py

@@ -4,7 +4,7 @@
 import logging
 
 from .._constants import _DB_TABLE_NAMES, _PHOTOS_4_VERSION
-from ..utils import _open_sql_file
+from ..utils import _open_sql_file, normalize_unicode
 from .photosdb_utils import get_db_version
 
 
@@ -121,7 +121,7 @@ def _process_faceinfo_4(photosdb):
         face["asset_uuid"] = asset_uuid
         face["uuid"] = row[2]
         face["person"] = person_id
-        face["fullname"] = row[3]
+        face["fullname"] = normalize_unicode(row[3])
         face["sourcewidth"] = row[7]
         face["sourceheight"] = row[8]
         face["centerx"] = row[9]
@@ -282,7 +282,7 @@ def _process_faceinfo_5(photosdb):
         face["asset_uuid"] = asset_uuid
         face["uuid"] = row[2]
         face["person"] = person_pk
-        face["fullname"] = row[4]
+        face["fullname"] = normalize_unicode(row[4])
         face["agetype"] = row[5]
         face["baldtype"] = row[6]
         face["eyemakeuptype"] = row[7]

osxphotos/photosdb/_photosdb_process_searchinfo.py

@@ -10,7 +10,7 @@ import uuid as uuidlib
 from pprint import pformat
 
 from .._constants import _PHOTOS_4_VERSION, SEARCH_CATEGORY_LABEL
-from ..utils import _db_is_locked, _debug, _open_sql_file
+from ..utils import _db_is_locked, _debug, _open_sql_file, normalize_unicode
 
 """
     This module should be imported in the class defintion of PhotosDB in photosdb.py
@@ -112,8 +112,8 @@ def _process_searchinfo(self):
         record["groupid"] = row[3]
         record["category"] = row[4]
         record["owning_groupid"] = row[5]
-        record["content_string"] = row[6].replace("\x00", "")
-        record["normalized_string"] = row[7].replace("\x00", "")
+        record["content_string"] = normalize_unicode(row[6].replace("\x00", ""))
+        record["normalized_string"] = normalize_unicode(row[7].replace("\x00", ""))
         record["lookup_identifier"] = row[8]
 
         try:
@@ -147,9 +147,10 @@ def _process_searchinfo(self):
             "_db_searchinfo_labels_normalized: \n"
             + pformat(self._db_searchinfo_labels_normalized)
         )
-    
+
     conn.close()
 
+
 @property
 def labels(self):
     """ return list of all search info labels found in the library """

osxphotos/photosdb/photosdb.py

@@ -8,7 +8,6 @@ import os
 import os.path
 import pathlib
 import platform
-import sqlite3
 import sys
 import tempfile
 from datetime import datetime, timedelta, timezone
@@ -26,15 +25,15 @@ from .._constants import (
     _PHOTOS_4_VERSION,
     _PHOTOS_5_ALBUM_KIND,
     _PHOTOS_5_FOLDER_KIND,
+    _PHOTOS_5_IMPORT_SESSION_ALBUM_KIND,
     _PHOTOS_5_ROOT_FOLDER_KIND,
     _PHOTOS_5_SHARED_ALBUM_KIND,
-    _PHOTOS_5_VERSION,
-    _TESTED_DB_VERSIONS,
     _TESTED_OS_VERSIONS,
     _UNKNOWN_PERSON,
+    TIME_DELTA,
 )
 from .._version import __version__
-from ..albuminfo import AlbumInfo, FolderInfo
+from ..albuminfo import AlbumInfo, FolderInfo, ImportInfo
 from ..datetime_utils import datetime_has_tz, datetime_naive_to_local
 from ..personinfo import PersonInfo
 from ..photoinfo import PhotoInfo
@@ -45,8 +44,9 @@ from ..utils import (
     _get_os_version,
     _open_sql_file,
     get_last_library_path,
+    normalize_unicode,
 )
-from .photosdb_utils import get_db_version, get_db_model_version
+from .photosdb_utils import get_db_model_version, get_db_version
 
 # TODO: Add test for imageTimeZoneOffsetSeconds = None
 # TODO: Add test for __str__
@@ -485,6 +485,18 @@ class PhotosDB:
             self._albums_shared = self._get_albums(shared=True)
             return self._albums_shared
 
+    @property
+    def import_info(self):
+        """ return list of ImportInfo objects for each import session in the database """
+        try:
+            return self._import_info
+        except AttributeError:
+            self._import_info = [
+                ImportInfo(db=self, uuid=album)
+                for album in self._get_album_uuids(import_session=True)
+            ]
+            return self._import_info
+
     @property
     def db_version(self):
         """ return the database version as stored in LiGlobals table """
@@ -514,6 +526,7 @@ class PhotosDB:
         """ If sqlite shared memory and write-ahead log files exist, those are copied too """
         # required because python's sqlite3 implementation can't read a locked file
         # _, suffix = os.path.splitext(fname)
+        dest_name = dest_path = ""
         try:
             dest_name = pathlib.Path(fname).name
             dest_path = os.path.join(self._tempdir_name, dest_name)
@@ -536,9 +549,6 @@ class PhotosDB:
         """ process the Photos database to extract info
             works on Photos version <= 4.0 """
 
-        # Epoch is Jan 1, 2001
-        td = (datetime(2001, 1, 1, 0, 0) - datetime(1970, 1, 1, 0, 0)).total_seconds()
-
         (conn, c) = _open_sql_file(self._tmp_db)
 
         # get info to associate persons with photos
@@ -685,7 +695,8 @@ class PhotosDB:
                 isInTrash, 
                 folderUuid,
                 albumType, 
-                albumSubclass 
+                albumSubclass,
+                createDate 
                 FROM RKAlbum """
         )
 
@@ -698,11 +709,12 @@ class PhotosDB:
         # 5:    folderUuid
         # 6:    albumType
         # 7:    albumSubclass -- if 3, normal user album
+        # 8:    createDate
 
         for album in c:
             self._dbalbum_details[album[0]] = {
                 "_uuid": album[0],
-                "title": album[1],
+                "title": normalize_unicode(album[1]),
                 "cloudlibrarystate": album[2],
                 "cloudidentifier": album[3],
                 "intrash": False if album[4] == 0 else True,
@@ -715,6 +727,9 @@ class PhotosDB:
                 "albumSubclass": album[7],
                 # for compatability with Photos 5 where album kind is ZKIND
                 "kind": album[7],
+                "creation_date": album[8],
+                "start_date": None,  # Photos 5 only
+                "end_date": None,  # Photos 5 only
             }
 
         # get details about folders
@@ -746,7 +761,7 @@ class PhotosDB:
             self._dbfolder_details[uuid] = {
                 "_uuid": row[0],
                 "modelId": row[1],
-                "name": row[2],
+                "name": normalize_unicode(row[2]),
                 "isMagic": row[3],
                 "intrash": row[4],
                 "folderType": row[5],
@@ -831,7 +846,8 @@ class PhotosDB:
                     RKMaster.height,
                     RKMaster.width, 
                     RKMaster.orientation,
-                    RKMaster.fileSize
+                    RKMaster.fileSize,
+                    RKVersion.subType
                     FROM RKVersion, RKMaster
                     WHERE RKVersion.masterUuid = RKMaster.uuid"""
             )
@@ -858,7 +874,8 @@ class PhotosDB:
                     RKMaster.height,
                     RKMaster.width, 
                     RKMaster.orientation,
-                    RKMaster.originalFileSize
+                    RKMaster.originalFileSize,
+                    RKVersion.subType
                     FROM RKVersion, RKMaster 
                     WHERE RKVersion.masterUuid = RKMaster.uuid"""
             )
@@ -904,6 +921,7 @@ class PhotosDB:
         # 37    RKMaster.width,
         # 38    RKMaster.orientation,
         # 39    RKMaster.originalFileSize
+        # 40    RKVersion.subType
 
         for row in c:
             uuid = row[0]
@@ -918,9 +936,10 @@ class PhotosDB:
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
+            self._dbphotos[uuid]["lastmodifieddate_timestamp"] = row[4]
             try:
                 self._dbphotos[uuid]["lastmodifieddate"] = datetime.fromtimestamp(
-                    row[4] + td
+                    row[4] + TIME_DELTA
                 )
             except ValueError:
                 self._dbphotos[uuid]["lastmodifieddate"] = None
@@ -928,9 +947,10 @@ class PhotosDB:
                 self._dbphotos[uuid]["lastmodifieddate"] = None
 
             self._dbphotos[uuid]["imageTimeZoneOffsetSeconds"] = row[9]
+            self._dbphotos[uuid]["imageDate_timestamp"] = row[5]
 
             try:
-                imagedate = datetime.fromtimestamp(row[5] + td)
+                imagedate = datetime.fromtimestamp(row[5] + TIME_DELTA)
                 seconds = self._dbphotos[uuid]["imageTimeZoneOffsetSeconds"] or 0
                 delta = timedelta(seconds=seconds)
                 tz = timezone(delta)
@@ -947,7 +967,7 @@ class PhotosDB:
             self._dbphotos[uuid]["volumeId"] = row[10]
             self._dbphotos[uuid]["imagePath"] = row[11]
             self._dbphotos[uuid]["extendedDescription"] = row[12]
-            self._dbphotos[uuid]["name"] = row[13]
+            self._dbphotos[uuid]["name"] = normalize_unicode(row[13])
             self._dbphotos[uuid]["isMissing"] = row[14]
             self._dbphotos[uuid]["originalFilename"] = row[15]
             self._dbphotos[uuid]["favorite"] = row[16]
@@ -972,6 +992,13 @@ class PhotosDB:
 
             self._dbphotos[uuid]["UTI"] = row[22]
 
+            # The UTI in RKMaster will always be UTI of the original
+            # Unlike Photos 5 which changes the UTI to match latest edit
+            self._dbphotos[uuid]["UTI_original"] = row[22]
+
+            # UTI edited will be read from RKModelResource
+            self._dbphotos[uuid]["UTI_edited"] = None
+
             # handle burst photos
             # if burst photo, determine whether or not it's a selected burst photo
             self._dbphotos[uuid]["burstUUID"] = row[23]
@@ -1038,11 +1065,6 @@ class PhotosDB:
             self._dbphotos[uuid]["cloudAvailable"] = None
             self._dbphotos[uuid]["incloud"] = None
 
-            # TODO: NOT YET USED -- PLACEHOLDER for RAW processing (currently only in _process_database5)
-            # original resource choice (e.g. RAW or jpeg)
-            self._dbphotos[uuid]["original_resource_choice"] = None
-            self._dbphotos[uuid]["raw_is_original"] = None
-
             # associated RAW image info
             self._dbphotos[uuid]["has_raw"] = True if row[25] == 7 else False
             self._dbphotos[uuid]["UTI_raw"] = None
@@ -1054,6 +1076,25 @@ class PhotosDB:
             self._dbphotos[uuid]["non_raw_master_uuid"] = row[30]
             self._dbphotos[uuid]["alt_master_uuid"] = row[31]
 
+            # original resource choice (e.g. RAW or jpeg)
+            # In Photos 5+, original_resource_choice set from:
+            # ZADDITIONALASSETATTRIBUTES.ZORIGINALRESOURCECHOICE
+            # = 0 if jpeg is selected as "original" in Photos (the default)
+            # = 1 if RAW is selected as "original" in Photos
+            # RKVersion.subType, RAW always appears to be 16
+            #   4 = mov
+            #   16 = RAW
+            #   32 = JPEG
+            #   64 = TIFF
+            #   2048 = PNG
+            #   32768 = HIEC
+            self._dbphotos[uuid]["original_resource_choice"] = (
+                1 if row[40] == 16 and self._dbphotos[uuid]["has_raw"] else 0
+            )
+            self._dbphotos[uuid]["raw_is_original"] = bool(
+                self._dbphotos[uuid]["original_resource_choice"]
+            )
+
             # recently deleted items
             self._dbphotos[uuid]["intrash"] = True if row[32] == 1 else False
 
@@ -1066,6 +1107,11 @@ class PhotosDB:
             self._dbphotos[uuid]["original_orientation"] = row[38]
             self._dbphotos[uuid]["original_filesize"] = row[39]
 
+            # import session not yet handled for Photos 4
+            self._dbphotos[uuid]["import_session"] = None
+            self._dbphotos[uuid]["import_uuid"] = None
+            self._dbphotos[uuid]["fok_import_session"] = None
+
         # get additional details from RKMaster, needed for RAW processing
         c.execute(
             """ SELECT 
@@ -1078,7 +1124,8 @@ class PhotosDB:
                 RKMaster.modelID, 
                 RKMaster.fileSize, 
                 RKMaster.isTrulyRaw,
-                RKMaster.alternateMasterUuid
+                RKMaster.alternateMasterUuid,
+                RKMaster.filename
                 FROM RKMaster
             """
         )
@@ -1094,6 +1141,7 @@ class PhotosDB:
         # 7     RKMaster.fileSize,
         # 8     RKMaster.isTrulyRaw,
         # 9     RKMaster.alternateMasterUuid
+        # 10    RKMaster.filename
 
         for row in c:
             uuid = row[0]
@@ -1108,6 +1156,7 @@ class PhotosDB:
             info["fileSize"] = row[7]
             info["isTrulyRAW"] = row[8]
             info["alternateMasterUuid"] = row[9]
+            info["filename"] = row[10]
             self._dbphotos_master[uuid] = info
 
         # get details needed to find path of the edited photos
@@ -1137,7 +1186,6 @@ class PhotosDB:
                     if (
                         row[1] != "UNADJUSTEDNONRAW"
                         and row[1] != "UNADJUSTED"
-                        # and row[4] == "public.jpeg"
                         and row[6] == 2
                     ):
                         if "edit_resource_id" in self._dbphotos[uuid]:
@@ -1151,6 +1199,7 @@ class PhotosDB:
                         # should we return all edits or just most recent one?
                         # For now, return most recent edit
                         self._dbphotos[uuid]["edit_resource_id"] = row[2]
+                        self._dbphotos[uuid]["UTI_edited"] = row[4]
 
         # get details on external edits
         c.execute(
@@ -1223,7 +1272,7 @@ class PhotosDB:
         )
 
         # Order of results
-        # 0  RKMaster.uuid,
+        # 0  RKVersion.uuid,
         # 1  RKMaster.cloudLibraryState,
         # 2  RKCloudResource.available,
         # 3  RKCloudResource.status
@@ -1274,11 +1323,12 @@ class PhotosDB:
 
             # get the place info that matches the RKPlace modelIDs for this photo
             # (place_ids), sort by area (element 3 of the place_data tuple in places)
+            # area could be None so assume 0 if it is (issue #230)
             place_names = [
                 pname
                 for pname in sorted(
                     [places[p] for p in places if p in place_ids],
-                    key=lambda place: place[3],
+                    key=lambda place: place[3] if place[3] is not None else 0,
                 )
             ]
 
@@ -1295,17 +1345,28 @@ class PhotosDB:
 
         # add volume name to _dbphotos_master
         for info in self._dbphotos_master.values():
-            info["volume"] = (
-                self._dbvolumes[info["volumeId"]]
-                if info["volumeId"] is not None
-                else None
-            )
+            # issue 230: have seen bad volumeID values
+            try:
+                info["volume"] = (
+                    self._dbvolumes[info["volumeId"]]
+                    if info["volumeId"] is not None
+                    else None
+                )
+            except KeyError:
+                info["volume"] = None
 
         # add data on RAW images
         for info in self._dbphotos.values():
             if info["has_raw"]:
                 raw_uuid = info["raw_master_uuid"]
                 info["raw_info"] = self._dbphotos_master[raw_uuid]
+                info["UTI_raw"] = self._dbphotos_master[raw_uuid]["UTI"]
+                non_raw_uuid = info["non_raw_master_uuid"]
+                info["raw_pair_info"] = self._dbphotos_master[non_raw_uuid]
+            else:
+                info["raw_info"] = None
+                info["UTI_raw"] = None
+                info["raw_pair_info"] = None
 
         # done with the database connection
         conn.close()
@@ -1336,9 +1397,13 @@ class PhotosDB:
                 self._dbphotos[uuid]["hasAlbums"] = 0
 
             if self._dbphotos[uuid]["volumeId"] is not None:
-                self._dbphotos[uuid]["volume"] = self._dbvolumes[
-                    self._dbphotos[uuid]["volumeId"]
-                ]
+                # issue 230: have seen bad volumeID values
+                try:
+                    self._dbphotos[uuid]["volume"] = self._dbvolumes[
+                        self._dbphotos[uuid]["volumeId"]
+                    ]
+                except KeyError:
+                    self._dbphotos[uuid]["volume"] = None
             else:
                 self._dbphotos[uuid]["volume"] = None
 
@@ -1419,16 +1484,17 @@ class PhotosDB:
         if _debug():
             logging.debug(f"_process_database5")
 
-        # Epoch is Jan 1, 2001
-        td = (datetime(2001, 1, 1, 0, 0) - datetime(1970, 1, 1, 0, 0)).total_seconds()
+        (conn, c) = _open_sql_file(self._tmp_db)
 
+        # some of the tables/columns have different names in different versions of Photos
         photos_ver = get_db_model_version(self._tmp_db)
         self._photos_ver = photos_ver
         asset_table = _DB_TABLE_NAMES[photos_ver]["ASSET"]
         keyword_join = _DB_TABLE_NAMES[photos_ver]["KEYWORD_JOIN"]
         album_join = _DB_TABLE_NAMES[photos_ver]["ALBUM_JOIN"]
-
-        (conn, c) = _open_sql_file(self._tmp_db)
+        album_sort = _DB_TABLE_NAMES[photos_ver]["ALBUM_SORT_ORDER"]
+        import_fok = _DB_TABLE_NAMES[photos_ver]["IMPORT_FOK"]
+        depth_state = _DB_TABLE_NAMES[photos_ver]["DEPTH_STATE"]
 
         # Look for all combinations of persons and pictures
         if _debug():
@@ -1539,7 +1605,7 @@ class PhotosDB:
             f""" SELECT 
                 ZGENERICALBUM.ZUUID, 
                 {asset_table}.ZUUID,
-                {album_join}
+                {album_sort}
                 FROM {asset_table} 
                 JOIN Z_26ASSETS ON {album_join} = {asset_table}.Z_PK 
                 JOIN ZGENERICALBUM ON ZGENERICALBUM.Z_PK = Z_26ASSETS.Z_26ALBUMS
@@ -1577,13 +1643,16 @@ class PhotosDB:
             "ZKIND, "  # 6
             "ZPARENTFOLDER, "  # 7
             "Z_PK, "  # 8
-            "ZTRASHEDSTATE "  # 9
+            "ZTRASHEDSTATE, "  # 9
+            "ZCREATIONDATE, "  # 10
+            "ZSTARTDATE, "  # 11
+            "ZENDDATE "  # 12
             "FROM ZGENERICALBUM "
         )
         for album in c:
             self._dbalbum_details[album[0]] = {
                 "_uuid": album[0],
-                "title": album[1],
+                "title": normalize_unicode(album[1]),
                 "cloudlocalstate": album[2],
                 "cloudownerfirstname": album[3],
                 "cloudownderlastname": album[4],
@@ -1594,6 +1663,9 @@ class PhotosDB:
                 "parentfolder": album[7],
                 "pk": album[8],
                 "intrash": False if album[9] == 0 else True,
+                "creation_date": album[10],
+                "start_date": album[11],
+                "end_date": album[12],
             }
 
             # add cross-reference by pk to uuid
@@ -1655,12 +1727,13 @@ class PhotosDB:
                 JOIN ZKEYWORD ON ZKEYWORD.Z_PK = {keyword_join} """
         )
         for keyword in c:
+            keyword_title = normalize_unicode(keyword[0])
             if not keyword[1] in self._dbkeywords_uuid:
                 self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword[0] in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword[0]] = []
+            if not keyword_title in self._dbkeywords_keyword:
+                self._dbkeywords_keyword[keyword_title] = []
             self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword[0]].append(keyword[1])
+            self._dbkeywords_keyword[keyword_title].append(keyword[1])
 
         if _debug():
             logging.debug(f"Finished walking through keywords")
@@ -1714,7 +1787,8 @@ class PhotosDB:
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALHEIGHT, 
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALWIDTH, 
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
-                ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE
+                ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE,
+                {depth_state}
                 FROM {asset_table} 
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = {asset_table}.Z_PK 
                 ORDER BY {asset_table}.ZUUID  """
@@ -1757,6 +1831,7 @@ class PhotosDB:
         # 33   ZADDITIONALASSETATTRIBUTES.ZORIGINALWIDTH,
         # 34   ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
         # 35   ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE
+        # 36   ZGENERICASSET.ZDEPTHSTATES / ZASSET.ZDEPTHTYPE
 
         for row in c:
             uuid = row[0]
@@ -1765,22 +1840,24 @@ class PhotosDB:
             info["modelID"] = None
             info["masterUuid"] = None
             info["masterFingerprint"] = row[1]
-            info["name"] = row[2]
+            info["name"] = normalize_unicode(row[2])
 
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
+            info["lastmodifieddate_timestamp"] = row[4]
             try:
-                info["lastmodifieddate"] = datetime.fromtimestamp(row[4] + td)
+                info["lastmodifieddate"] = datetime.fromtimestamp(row[4] + TIME_DELTA)
             except ValueError:
                 info["lastmodifieddate"] = None
             except TypeError:
                 info["lastmodifieddate"] = None
 
             info["imageTimeZoneOffsetSeconds"] = row[6]
+            info["imageDate_timestamp"] = row[5]
 
             try:
-                imagedate = datetime.fromtimestamp(row[5] + td)
+                imagedate = datetime.fromtimestamp(row[5] + TIME_DELTA)
                 seconds = info["imageTimeZoneOffsetSeconds"] or 0
                 delta = timedelta(seconds=seconds)
                 tz = timezone(delta)
@@ -1876,10 +1953,10 @@ class PhotosDB:
             # 3 = HDR photo
             # 4 = non-HDR version of the photo
             # 6 = panorama
-            # 8 = portrait
+            # > 6 = portrait (sometimes, see ZDEPTHSTATE/ZDEPTHTYPE)
             info["customRenderedValue"] = row[22]
             info["hdr"] = True if row[22] == 3 else False
-            info["portrait"] = True if row[22] == 8 else False
+            info["portrait"] = True if row[36] != 0 else False
 
             # Set panorama from either KindSubType or RenderedValue
             info["panorama"] = True if row[21] == 1 or row[22] == 6 else False
@@ -1925,6 +2002,12 @@ class PhotosDB:
             info["original_orientation"] = row[34]
             info["original_filesize"] = row[35]
 
+            # initialize import session info which will be filled in later
+            # not every photo has an import session so initialize all records now
+            info["import_session"] = None
+            info["fok_import_session"] = None
+            info["import_uuid"] = None
+
             # associated RAW image info
             # will be filled in later
             info["has_raw"] = False
@@ -1951,6 +2034,32 @@ class PhotosDB:
             # else:
             #     info["burst"] = False
 
+        # get info on import sessions
+        # 0    ZGENERICASSET.ZUUID
+        # 1    ZGENERICASSET.ZIMPORTSESSION
+        # 2    ZGENERICASSET.Z_FOK_IMPORTSESSION
+        # 3    ZGENERICALBUM.ZUUID,
+        c.execute(
+            f"""SELECT
+                {asset_table}.ZUUID,
+                {asset_table}.ZIMPORTSESSION,
+                {import_fok},
+                ZGENERICALBUM.ZUUID
+                FROM
+                {asset_table}
+                JOIN ZGENERICALBUM ON ZGENERICALBUM.Z_PK = {asset_table}.ZIMPORTSESSION
+            """
+        )
+
+        for row in c:
+            uuid = row[0]
+            try:
+                self._dbphotos[uuid]["import_session"] = row[1]
+                self._dbphotos[uuid]["fok_import_session"] = row[2]
+                self._dbphotos[uuid]["import_uuid"] = row[3]
+            except KeyError:
+                logging.debug(f"No info record for uuid {uuid} for import session")
+
         # Get extended description
         c.execute(
             f"""SELECT {asset_table}.ZUUID, 
@@ -1963,7 +2072,7 @@ class PhotosDB:
         for row in c:
             uuid = row[0]
             if uuid in self._dbphotos:
-                self._dbphotos[uuid]["extendedDescription"] = row[1]
+                self._dbphotos[uuid]["extendedDescription"] = normalize_unicode(row[1])
             else:
                 if _debug():
                     logging.debug(
@@ -1995,36 +2104,43 @@ class PhotosDB:
         # determine if a photo is missing in Photos 5
 
         # Get info on remote/local availability for photos in shared albums
+        # Also get UTI of original image (zdatastoresubtype = 1)
         c.execute(
             f""" SELECT 
                 {asset_table}.ZUUID, 
                 ZINTERNALRESOURCE.ZLOCALAVAILABILITY, 
-                ZINTERNALRESOURCE.ZREMOTEAVAILABILITY
+                ZINTERNALRESOURCE.ZREMOTEAVAILABILITY,
+                ZINTERNALRESOURCE.ZDATASTORESUBTYPE,
+                ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER,
+                ZUNIFORMTYPEIDENTIFIER.ZIDENTIFIER
                 FROM {asset_table}
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = {asset_table}.Z_PK 
                 JOIN ZINTERNALRESOURCE ON ZINTERNALRESOURCE.ZASSET = ZADDITIONALASSETATTRIBUTES.ZASSET 
+                JOIN ZUNIFORMTYPEIDENTIFIER ON ZUNIFORMTYPEIDENTIFIER.Z_PK = ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER 
                 WHERE  ZDATASTORESUBTYPE = 1 OR ZDATASTORESUBTYPE = 3 """
         )
 
+        # Order of results:
+        # 0 {asset_table}.ZUUID,
+        # 1 ZINTERNALRESOURCE.ZLOCALAVAILABILITY,
+        # 2 ZINTERNALRESOURCE.ZREMOTEAVAILABILITY,
+        # 3 ZINTERNALRESOURCE.ZDATASTORESUBTYPE,
+        # 4 ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER,
+        # 5 ZUNIFORMTYPEIDENTIFIER.ZIDENTIFIER
+
         for row in c:
             uuid = row[0]
             if uuid in self._dbphotos:
-                #  and self._dbphotos[uuid]["isMissing"] is None:
                 self._dbphotos[uuid]["localAvailability"] = row[1]
                 self._dbphotos[uuid]["remoteAvailability"] = row[2]
-
-                # old = self._dbphotos[uuid]["isMissing"]
+                if row[3] == 1:
+                    self._dbphotos[uuid]["UTI_original"] = row[5]
 
                 if row[1] != 1:
                     self._dbphotos[uuid]["isMissing"] = 1
                 else:
                     self._dbphotos[uuid]["isMissing"] = 0
 
-                # if old is not None and old != self._dbphotos[uuid]["isMissing"]:
-                #     logging.warning(
-                #         f"{uuid} isMissing changed: {old} {self._dbphotos[uuid]['isMissing']}"
-                #     )
-
         # get information on local/remote availability
         c.execute(
             f""" SELECT {asset_table}.ZUUID,
@@ -2041,18 +2157,11 @@ class PhotosDB:
                 self._dbphotos[uuid]["localAvailability"] = row[1]
                 self._dbphotos[uuid]["remoteAvailability"] = row[2]
 
-                # old = self._dbphotos[uuid]["isMissing"]
-
                 if row[1] != 1:
                     self._dbphotos[uuid]["isMissing"] = 1
                 else:
                     self._dbphotos[uuid]["isMissing"] = 0
 
-                # if old is not None and old != self._dbphotos[uuid]["isMissing"]:
-                #     logging.warning(
-                #         f"{uuid} isMissing changed: {old} {self._dbphotos[uuid]['isMissing']}"
-                #     )
-
         # get information about cloud sync state
         c.execute(
             f""" SELECT
@@ -2362,16 +2471,26 @@ class PhotosDB:
         hierarchy = _recurse_folder_hierarchy(folders)
         return hierarchy
 
-    def _get_album_uuids(self, shared=False):
+    def _get_album_uuids(self, shared=False, import_session=False):
         """ Return list of album UUIDs found in photos database
         
             Filters out albums in the trash and any special album types
         
         Args:
             shared: boolean; if True, returns shared albums, else normal albums
+            import_session: boolean, if True, returns import session albums, else normal or shared albums
+            Note: flags (shared, import_session) are mutually exclusive
         
+        Raises:
+            ValueError: raised if mutually exclusive flags passed
+
         Returns: list of album UUIDs 
         """
+        if shared and import_session:
+            raise ValueError(
+                "flags are mutually exclusive: pass zero or one of shared, import_session"
+            )
+
         if self._db_version <= _PHOTOS_4_VERSION:
             version4 = True
             if shared:
@@ -2379,11 +2498,21 @@ class PhotosDB:
                     f"Shared albums not implemented for Photos library version {self._db_version}"
                 )
                 return []  # not implemented for _PHOTOS_4_VERSION
+            elif import_session:
+                logging.warning(
+                    f"Import sessions not implemented for Photos library version {self._db_version}"
+                )
+                return []  # not implemented for _PHOTOS_4_VERSION
             else:
                 album_kind = _PHOTOS_4_ALBUM_KIND
         else:
             version4 = False
-            album_kind = _PHOTOS_5_SHARED_ALBUM_KIND if shared else _PHOTOS_5_ALBUM_KIND
+            if shared:
+                album_kind = _PHOTOS_5_SHARED_ALBUM_KIND
+            elif import_session:
+                album_kind = _PHOTOS_5_IMPORT_SESSION_ALBUM_KIND
+            else:
+                album_kind = _PHOTOS_5_ALBUM_KIND
 
         album_list = []
         # look through _dbalbum_details because _dbalbums_album won't have empty albums it

osxphotos/phototemplate.py

@@ -621,6 +621,9 @@ class PhotoTemplate:
         """ return list of values for a multi-valued template field """
         if field == "album":
             values = self.photo.albums
+            values = [
+                value.replace("/", ":") for value in values
+            ]  # TODO: temp fix for issue #213
         elif field == "keyword":
             values = self.photo.keywords
         elif field == "person":
@@ -638,11 +641,13 @@ class PhotoTemplate:
                 if album.folder_names:
                     # album in folder
                     folder = path_sep.join(album.folder_names)
-                    folder += path_sep + album.title
+                    folder += path_sep + album.title.replace(
+                        "/", ":"
+                    )  # TODO: temp fix for issue #213
                     values.append(folder)
                 else:
                     # album not in folder
-                    values.append(album.title)
+                    values.append(album.title.replace("/", ":"))
         else:
             raise ValueError(f"Unhandleded template value: {field}")
 

osxphotos/placeinfo.py

@@ -11,6 +11,9 @@ from collections import namedtuple  # pylint: disable=syntax-error
 import yaml
 from bpylist import archiver
 
+from ._constants import UNICODE_FORMAT
+from .utils import normalize_unicode
+
 # postal address information, returned by PlaceInfo.address
 PostalAddress = namedtuple(
     "PostalAddress",
@@ -76,12 +79,12 @@ class PLRevGeoLocationInfo:
         geoServiceProvider,
         postalAddress,
     ):
-        self.addressString = addressString
+        self.addressString = normalize_unicode(addressString)
         self.countryCode = countryCode
         self.mapItem = mapItem
         self.isHome = isHome
-        self.compoundNames = compoundNames
-        self.compoundSecondaryNames = compoundSecondaryNames
+        self.compoundNames = normalize_unicode(compoundNames)
+        self.compoundSecondaryNames = normalize_unicode(compoundSecondaryNames)
         self.version = version
         self.geoServiceProvider = geoServiceProvider
         self.postalAddress = postalAddress
@@ -183,7 +186,7 @@ class PLRevGeoMapItemAdditionalPlaceInfo:
 
     def __init__(self, area, name, placeType, dominantOrderType):
         self.area = area
-        self.name = name
+        self.name = normalize_unicode(name)
         self.placeType = placeType
         self.dominantOrderType = dominantOrderType
 
@@ -232,13 +235,13 @@ class CNPostalAddress:
         _subLocality,
     ):
         self._ISOCountryCode = _ISOCountryCode
-        self._city = _city
-        self._country = _country
-        self._postalCode = _postalCode
-        self._state = _state
-        self._street = _street
-        self._subAdministrativeArea = _subAdministrativeArea
-        self._subLocality = _subLocality
+        self._city = normalize_unicode(_city)
+        self._country = normalize_unicode(_country)
+        self._postalCode = normalize_unicode(_postalCode)
+        self._state = normalize_unicode(_state)
+        self._street = normalize_unicode(_street)
+        self._subAdministrativeArea = normalize_unicode(_subAdministrativeArea)
+        self._subLocality = normalize_unicode(_subLocality)
 
     def __eq__(self, other):
         return all(
@@ -414,9 +417,9 @@ class PlaceInfo4(PlaceInfo):
             #   2: type
             #   3: area
             try:
-                places_dict[p[2]].append((p[1], p[3]))
+                places_dict[p[2]].append((normalize_unicode(p[1]), p[3]))
             except KeyError:
-                places_dict[p[2]] = [(p[1], p[3])]
+                places_dict[p[2]] = [(normalize_unicode(p[1]), p[3])]
 
         # build list to populate PlaceNames tuple
         # initialize with empty lists for each field in PlaceNames

osxphotos/templates/xmp_sidecar.mako

@@ -1,5 +1,13 @@
 <!-- Created with osxphotos https://github.com/RhetTbull/osxphotos -->
 
+<%def name="photoshop_sidecar_for_extension(extension)">
+    % if extension is None:
+        <photoshop:SidecarForExtension></photoshop:SidecarForExtension>
+    % else:
+        <photoshop:SidecarForExtension>${extension}</photoshop:SidecarForExtension>
+    % endif
+</%def>
+
 <%def name="dc_description(desc)">
     % if desc is None:
         <dc:description></dc:description>
@@ -86,6 +94,7 @@
         <rdf:Description rdf:about="" 
             xmlns:dc="http://purl.org/dc/elements/1.1/" 
             xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
+            ${photoshop_sidecar_for_extension(extension)}
             ${dc_description(description)}
             ${dc_title(photo.title)}
             ${dc_subject(subjects)}

osxphotos/utils.py

@@ -10,6 +10,7 @@ import sqlite3
 import subprocess
 import sys
 import tempfile
+import unicodedata
 import urllib.parse
 from plistlib import load as plistload
 
@@ -18,6 +19,7 @@ import CoreServices
 import objc
 from Foundation import *
 
+from ._constants import UNICODE_FORMAT
 from .fileutil import FileUtil
 
 _DEBUG = False
@@ -352,3 +354,13 @@ def _db_is_locked(dbname):
 #     attr = xattr.xattr(filepath)
 #     uuid_bytes = bytes(uuid, 'utf-8')
 #     attr.set(OSXPHOTOS_XATTR_UUID, uuid_bytes)
+
+
+def normalize_unicode(value):
+    """ normalize unicode data """
+    if value is not None:
+        if not isinstance(value, str):
+            raise ValueError("value must be str")
+        return unicodedata.normalize(UNICODE_FORMAT, value)
+    else:
+        return None

requirements.txt

@@ -202,5 +202,6 @@ virtualenv==20.0.30
 wcwidth==0.1.9
 webencodings==0.5.1
 wrapt==1.11.1
+wurlitzer==2.0.1
 yarl==1.4.2
-zipp==0.5.2
+zipp==0.5.2

setup.py

@@ -78,6 +78,7 @@ setup(
         "bpylist2==3.0.2",
         "pathvalidate==2.2.1",
         "dataclasses==0.7;python_version<'3.7'",
+        "wurlitzer>=2.0.1",
     ],
     entry_points={"console_scripts": ["osxphotos=osxphotos.__main__:cli"]},
     include_package_data=True,

tests/README.md

@@ -17,6 +17,9 @@ Some of the export tests rely on photos in my local library and will look for `O
 
 One test for locale does not run on GitHub's automated workflow and will look for `OSXPHOTOS_TEST_LOCALE=1` to determine if it should be run.  If you want to run this test, set the environment variable.  
 
+## Test Photo Libraries
+**Important**: The test code uses several test photo libraries created on various version of MacOS.  If you need to inspect one of these or modify one for a test, make a copy of the library (for example, copy it to your ~/Pictures folder) then open the copy in Photos.  Once done, copy the revised library back to the tests/ folder.  If you do not do this, the Photos background process photoanalysisd will forever try to process the library resulting in updates to the database which will cause git to see changes to the file you didn't intend.  I'm not aware of any way to disassociate photoanalysisd from the library once you've opened it in Photos.
+
 ## Attribution ##
 These tests utilize a test Photos library. The test library is populated with photos from [flickr](https://www.flickr.com) and from my own photo library.  All images used are licensed under Creative Commons 2.0 Attribution [license](https://creativecommons.org/licenses/by/2.0/).  
 

tests/Test-10.14.6.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -36,7 +36,7 @@
 			<key>other</key>
 			<integer>0</integer>
 			<key>photos</key>
-			<integer>6</integer>
+			<integer>11</integer>
 			<key>videos</key>
 			<integer>0</integer>
 		</dict>

tests/Test-10.14.6.photoslibrary/private/com.apple.photoanalysisd/GraphService/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-07-27T03:16:28Z</date>
+	<date>2020-10-09T16:14:42Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-07-27T12:35:43Z</date>
+	<date>2020-10-10T05:21:03Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/private/com.apple.photomodel/appPrivateData.plist

@@ -5,7 +5,7 @@
 	<key>LithiumMessageTracer</key>
 	<dict>
 		<key>LastReportedDate</key>
-		<date>2020-07-27T03:18:40Z</date>
+		<date>2020-10-04T23:49:39Z</date>
 	</dict>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/moments/analysismetadata.plist

@@ -11,6 +11,6 @@
 	<key>PLLastRevGeoForcedProviderOutOfDateCheckVersionKey</key>
 	<integer>1</integer>
 	<key>PLLastRevGeoVerFileFetchDateKey</key>
-	<date>2020-07-27T03:16:25Z</date>
+	<date>2020-10-04T23:43:17Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/moments/historicalmarker.plist

@@ -3,7 +3,7 @@
 <plist version="1.0">
 <dict>
 	<key>LastHistoryRowId</key>
-	<integer>707</integer>
+	<integer>948</integer>
 	<key>LibraryBuildTag</key>
 	<string>D8C4AAA1-3AB6-4A65-BEBD-99CC3E5D433E</string>
 	<key>LibrarySchemaVersion</key>

tests/Test-10.14.6.photoslibrary/resources/recovery/Info.plist

@@ -9,7 +9,7 @@
 	<key>HistoricalMarker</key>
 	<dict>
 		<key>LastHistoryRowId</key>
-		<integer>707</integer>
+		<integer>948</integer>
 		<key>LibraryBuildTag</key>
 		<string>D8C4AAA1-3AB6-4A65-BEBD-99CC3E5D433E</string>
 		<key>LibrarySchemaVersion</key>
@@ -24,7 +24,7 @@
 	<key>SnapshotCompletedDate</key>
 	<date>2019-07-27T13:16:43Z</date>
 	<key>SnapshotLastValidated</key>
-	<date>2020-07-27T03:18:40Z</date>
+	<date>2020-10-10T05:22:36Z</date>
 	<key>SnapshotTables</key>
 	<dict/>
 </dict>

tests/Test-10.15.6.photoslibrary/database/DataModelVersion.plist

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>LibrarySchemaVersion</key>
+	<integer>5001</integer>
+	<key>MetaSchemaVersion</key>
+	<integer>3</integer>
+</dict>
+</plist>

tests/Test-10.15.6.photoslibrary/database/Photos.sqlite.lock

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>hostname</key>
+	<string>Rhets-MacBook-Pro.local</string>
+	<key>hostuuid</key>
+	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
+	<key>pid</key>
+	<integer>2964</integer>
+	<key>processname</key>
+	<string>photolibraryd</string>
+	<key>uid</key>
+	<integer>501</integer>
+</dict>
+</plist>

tests/Test-10.15.6.photoslibrary/database/search/searchMetadata.plist

@@ -0,0 +1,188 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BlacklistedMeaningsByMeaning</key>
+	<dict/>
+	<key>MePersonUUID</key>
+	<string>39488755-78C0-40B2-B378-EDA280E1823C</string>
+	<key>SceneWhitelist</key>
+	<array>
+		<string>Graduation</string>
+		<string>Aquarium</string>
+		<string>Food</string>
+		<string>Ice Skating</string>
+		<string>Mountain</string>
+		<string>Cliff</string>
+		<string>Basketball</string>
+		<string>Tennis</string>
+		<string>Jewelry</string>
+		<string>Cheese</string>
+		<string>Softball</string>
+		<string>Football</string>
+		<string>Circus</string>
+		<string>Jet Ski</string>
+		<string>Playground</string>
+		<string>Carousel</string>
+		<string>Paint Ball</string>
+		<string>Windsurfing</string>
+		<string>Sailboat</string>
+		<string>Sunbathing</string>
+		<string>Dam</string>
+		<string>Fireplace</string>
+		<string>Flower</string>
+		<string>Scuba</string>
+		<string>Hiking</string>
+		<string>Cetacean</string>
+		<string>Pier</string>
+		<string>Bowling</string>
+		<string>Snowboarding</string>
+		<string>Zoo</string>
+		<string>Snowmobile</string>
+		<string>Theater</string>
+		<string>Boat</string>
+		<string>Casino</string>
+		<string>Car</string>
+		<string>Diving</string>
+		<string>Cycling</string>
+		<string>Musical Instrument</string>
+		<string>Board Game</string>
+		<string>Castle</string>
+		<string>Sunset Sunrise</string>
+		<string>Martial Arts</string>
+		<string>Motocross</string>
+		<string>Submarine</string>
+		<string>Cat</string>
+		<string>Snow</string>
+		<string>Kiteboarding</string>
+		<string>Squash</string>
+		<string>Geyser</string>
+		<string>Music</string>
+		<string>Archery</string>
+		<string>Desert</string>
+		<string>Blackjack</string>
+		<string>Fireworks</string>
+		<string>Sportscar</string>
+		<string>Feline</string>
+		<string>Soccer</string>
+		<string>Museum</string>
+		<string>Baby</string>
+		<string>Fencing</string>
+		<string>Railroad</string>
+		<string>Nascar</string>
+		<string>Sky Surfing</string>
+		<string>Bird</string>
+		<string>Games</string>
+		<string>Baseball</string>
+		<string>Dressage</string>
+		<string>Snorkeling</string>
+		<string>Pyramid</string>
+		<string>Kite</string>
+		<string>Rowboat</string>
+		<string>Golf</string>
+		<string>Watersports</string>
+		<string>Lightning</string>
+		<string>Canyon</string>
+		<string>Auditorium</string>
+		<string>Night Sky</string>
+		<string>Karaoke</string>
+		<string>Skiing</string>
+		<string>Parade</string>
+		<string>Forest</string>
+		<string>Hot Air Balloon</string>
+		<string>Dragon Parade</string>
+		<string>Easter Egg</string>
+		<string>Monument</string>
+		<string>Jungle</string>
+		<string>Thanksgiving</string>
+		<string>Jockey Horse</string>
+		<string>Stadium</string>
+		<string>Airplane</string>
+		<string>Ballet</string>
+		<string>Yoga</string>
+		<string>Coral Reef</string>
+		<string>Skating</string>
+		<string>Wrestling</string>
+		<string>Bicycle</string>
+		<string>Tattoo</string>
+		<string>Amusement Park</string>
+		<string>Canoe</string>
+		<string>Cheerleading</string>
+		<string>Ping Pong</string>
+		<string>Fishing</string>
+		<string>Magic</string>
+		<string>Reptile</string>
+		<string>Winter Sport</string>
+		<string>Waterfall</string>
+		<string>Train</string>
+		<string>Bonsai</string>
+		<string>Surfing</string>
+		<string>Dog</string>
+		<string>Cake</string>
+		<string>Sledding</string>
+		<string>Sandcastle</string>
+		<string>Glacier</string>
+		<string>Lighthouse</string>
+		<string>Equestrian</string>
+		<string>Rafting</string>
+		<string>Shore</string>
+		<string>Hockey</string>
+		<string>Santa Claus</string>
+		<string>Formula One Car</string>
+		<string>Sport</string>
+		<string>Vehicle</string>
+		<string>Boxing</string>
+		<string>Rollerskating</string>
+		<string>Underwater</string>
+		<string>Orchestra</string>
+		<string>Carnival</string>
+		<string>Rocket</string>
+		<string>Skateboarding</string>
+		<string>Helicopter</string>
+		<string>Performance</string>
+		<string>Oktoberfest</string>
+		<string>Water Polo</string>
+		<string>Skate Park</string>
+		<string>Animal</string>
+		<string>Nightclub</string>
+		<string>String Instrument</string>
+		<string>Dinosaur</string>
+		<string>Gymnastics</string>
+		<string>Cricket</string>
+		<string>Volcano</string>
+		<string>Lake</string>
+		<string>Aurora</string>
+		<string>Dancing</string>
+		<string>Concert</string>
+		<string>Rock Climbing</string>
+		<string>Hang Glider</string>
+		<string>Rodeo</string>
+		<string>Fish</string>
+		<string>Art</string>
+		<string>Motorcycle</string>
+		<string>Volleyball</string>
+		<string>Wake Boarding</string>
+		<string>Badminton</string>
+		<string>Motor Sport</string>
+		<string>Sumo</string>
+		<string>Parasailing</string>
+		<string>Skydiving</string>
+		<string>Kickboxing</string>
+		<string>Pinata</string>
+		<string>Foosball</string>
+		<string>Go Kart</string>
+		<string>Poker</string>
+		<string>Kayak</string>
+		<string>Swimming</string>
+		<string>Atv</string>
+		<string>Beach</string>
+		<string>Dartboard</string>
+		<string>Athletics</string>
+		<string>Camping</string>
+		<string>Tornado</string>
+		<string>Billiards</string>
+		<string>Rugby</string>
+		<string>Airshow</string>
+	</array>
+</dict>
+</plist>

tests/Test-10.15.6.photoslibrary/database/search/searchProgress.plist

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>insertAlbum</key>
+	<array/>
+	<key>insertAsset</key>
+	<array/>
+	<key>insertHighlight</key>
+	<array/>
+	<key>insertMemory</key>
+	<array/>
+	<key>insertMoment</key>
+	<array/>
+	<key>removeAlbum</key>
+	<array/>
+	<key>removeAsset</key>
+	<array/>
+	<key>removeHighlight</key>
+	<array/>
+	<key>removeMemory</key>
+	<array/>
+	<key>removeMoment</key>
+	<array/>
+</dict>
+</plist>

tests/Test-10.15.6.photoslibrary/database/search/searchSystemInfo.plist

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>embeddingVersion</key>
+	<string>1</string>
+	<key>localeIdentifier</key>
+	<string>en_US</string>
+	<key>sceneTaxonomySHA</key>
+	<string>87914a047c69fbe8013fad2c70fa70c6c03b08b56190fe4054c880e6b9f57cc3</string>
+	<key>searchIndexVersion</key>
+	<string>10</string>
+</dict>
+</plist>