Fixed handling of date_modified for Catalina, issue #247

Convert to jpeg

CHANGELOG.md

@@ -4,16 +4,88 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.36.0](https://github.com/RhetTbull/osxphotos/compare/v0.35.7...v0.36.0)
+
+> 26 October 2020
+
+- Added verbose to PhotosDB(), partial fix for #110 [`d87b8f3`](https://github.com/RhetTbull/osxphotos/commit/d87b8f30a45cbb6fdb315a12f8585e2bdc21be6b)
+- Added comments/likes, implements #214 [`23de6b5`](https://github.com/RhetTbull/osxphotos/commit/23de6b58908371d9ca55d1d1999c6d56de454180)
+- Cleaned up constructor for PhotosDB [`667c89e`](https://github.com/RhetTbull/osxphotos/commit/667c89e32c3f96baeafebc03e83517ea05693b00)
+
+#### [v0.35.7](https://github.com/RhetTbull/osxphotos/compare/v0.35.6...v0.35.7)
+
+> 24 October 2020
+
+- Fix for issue #238 [`48f29e1`](https://github.com/RhetTbull/osxphotos/commit/48f29e138e4e9da3eba78f3681ee9b8cb28910df)
+
+#### [v0.35.6](https://github.com/RhetTbull/osxphotos/compare/v0.35.5...v0.35.6)
+
+> 24 October 2020
+
+- Fixed shared, not_shared in cli [`8551981`](https://github.com/RhetTbull/osxphotos/commit/8551981f68f0cd2a3a081cc21ae287ff981b9b4b)
+
+#### [v0.35.5](https://github.com/RhetTbull/osxphotos/compare/v0.35.4...v0.35.5)
+
+> 22 October 2020
+
+- Added get_shared_photo_comments.py to examples [`15e0914`](https://github.com/RhetTbull/osxphotos/commit/15e0914af6301a945bc751173aef6718487d9637)
+- Fix for issue #237 [`a416de2`](https://github.com/RhetTbull/osxphotos/commit/a416de29e4ac39a5c323f7913b05a8c38ad205be)
+- Added test for issue #235 [`ea68229`](https://github.com/RhetTbull/osxphotos/commit/ea68229ddac2e2301ac2d5607451cf7d00207d5d)
+
+#### [v0.35.4](https://github.com/RhetTbull/osxphotos/compare/v0.35.3...v0.35.4)
+
+> 18 October 2020
+
+- refactored template code to fix #213 [`#213`](https://github.com/RhetTbull/osxphotos/issues/213)
+
+#### [v0.35.3](https://github.com/RhetTbull/osxphotos/compare/v0.35.2...v0.35.3)
+
+> 15 October 2020
+
+- Fix for issue #235, #236 [`41b2399`](https://github.com/RhetTbull/osxphotos/commit/41b23991df3d1d553b70889ede237f83b6874519)
+
+#### [v0.35.2](https://github.com/RhetTbull/osxphotos/compare/v0.35.1...v0.35.2)
+
+> 12 October 2020
+
+- Fix for issue #234 [`da100f9`](https://github.com/RhetTbull/osxphotos/commit/da100f93a9b849ca4750336d7f90e9023e39dd07)
+
+#### [v0.35.1](https://github.com/RhetTbull/osxphotos/compare/v0.35.0...v0.35.1)
+
+> 12 October 2020
+
+- Fix for issue #230 [`dcbf8f2`](https://github.com/RhetTbull/osxphotos/commit/dcbf8f25f61e21bcf1040046aa9d6ddba4ac9735)
+
+#### [v0.35.0](https://github.com/RhetTbull/osxphotos/compare/v0.34.5...v0.35.0)
+
+> 12 October 2020
+
+- Convert to jpeg [`#233`](https://github.com/RhetTbull/osxphotos/pull/233)
+- Updated tests, closes #231 [`#231`](https://github.com/RhetTbull/osxphotos/issues/231)
+- Updated tests [`b0171ba`](https://github.com/RhetTbull/osxphotos/commit/b0171ba6f5b73e1ff71e16d27852f8df7f208f60)
+- Updated tests [`07b0843`](https://github.com/RhetTbull/osxphotos/commit/07b08433df5a60f191e23a95394e83e51dca016f)
+- Merge branch 'master' into convert_to_jpeg [`fe5185b`](https://github.com/RhetTbull/osxphotos/commit/fe5185be8893002da663039f8ec103faed0f1831)
+- Added israw, tests for Big Sur [`b5a9794`](https://github.com/RhetTbull/osxphotos/commit/b5a9794f6bff5683fd42a22197454940e4d7ba88)
+- Updates to path, path_raw, uti for RAW+JPEG pairs [`b32f4b8`](https://github.com/RhetTbull/osxphotos/commit/b32f4b8504768a5f4b5ad54c00315b9e82fca980)
+
+#### [v0.34.5](https://github.com/RhetTbull/osxphotos/compare/v0.34.3...v0.34.5)
+
+> 6 October 2020
+
+- --convert-to-jpeg initial version working [`38f201d`](https://github.com/RhetTbull/osxphotos/commit/38f201d0fb70bf299a828c1dd0d034a119e380c4)
+- Added tests, fixed bug in export_db [`5a13605`](https://github.com/RhetTbull/osxphotos/commit/5a13605f850bb947c8888246f06a5ca4e6aa5f10)
+- Updated tests [`b2b39aa`](https://github.com/RhetTbull/osxphotos/commit/b2b39aa6075df11861cf5d8945b657204f120e87)
+
 #### [v0.34.3](https://github.com/RhetTbull/osxphotos/compare/v0.34.2...v0.34.3)
 
 > 29 September 2020
 
 - Update exiftool.py to preserve file modification time, thanks to @hhoeck  [`#223`](https://github.com/RhetTbull/osxphotos/pull/223)
 - Added tests for 10.15.6 [`432da7f`](https://github.com/RhetTbull/osxphotos/commit/432da7f139a5e4b37eeb358f4ede45314407f8e5)
+- Added HEIC test image [`ddc1e69`](https://github.com/RhetTbull/osxphotos/commit/ddc1e69b4a4ac712e1af312b865c4216f9ad350c)
 - Fixed bug related to issue #222 [`c939df7`](https://github.com/RhetTbull/osxphotos/commit/c939df717159e8b97955c0b267327cd56a9ed56c)
 - Version bump for bug fix [`62d54cc`](https://github.com/RhetTbull/osxphotos/commit/62d54cc0beabd0141545608184d4b2c658eedf0f)
 - Update README.md [`6883fec`](https://github.com/RhetTbull/osxphotos/commit/6883fec2b2236d892b88327e1b4e9da1237f7dea)
-- Update exiftool.py [`3d21dad`](https://github.com/RhetTbull/osxphotos/commit/3d21dadf4102e9101e48a0c6f739a544f7f9d9de)
 
 #### [v0.34.2](https://github.com/RhetTbull/osxphotos/compare/v0.34.1...v0.34.2)
 

README.md

@@ -21,6 +21,9 @@
     + [ScoreInfo](#scoreinfo)
     + [PersonInfo](#personinfo)
     + [FaceInfo](#faceinfo)
+    + [CommentInfo](#commentinfo)
+    + [LikeInfo](#likeinfo)
+    + [Raw Photos](#raw-photos)
     + [Template Substitutions](#template-substitutions)
     + [Utility Functions](#utility-functions)
   * [Examples](#examples)
@@ -57,18 +60,23 @@ You can also install directly from [pypi](https://pypi.org/project/osxphotos/):
 
     pip install osxphotos
 
-**WARNING** The git repo for this project is very large (> 1GB) because it contains multiple Photos libraries used for testing on different versions of MacOS.  If you just want to use the osxphotos package in your own code, I recommend you install the latest version from [PyPI](https://pypi.org/project/osxphotos/). If you just want to use the command line utility, you can download a pre-built executable of the latest [release](https://github.com/RhetTbull/osxphotos/releases) or you can install via `pip` which also installs the command line app.  If you aren't comfortable with running python on your Mac, start with the pre-built executable.
+I recommend you create a [virtual environment](https://docs.python.org/3/tutorial/venv.html) before installing osxphotos.
+
+If you aren't familiar with installing python applications, I recommend you install `osxphotos` with [pipx](https://github.com/pipxproject/pipx). If you use `pipx`, you will not need to create a virtual environment as `pipx` takes care of this. The easiest way to do this on a Mac is to use [homebrew](https://brew.sh/):
+
+- Open `Terminal` (search for `Terminal` in Spotlight or look in `Applications/Utilities`)
+- Install `homebrew` according to instructions at [https://brew.sh/](https://brew.sh/)
+- Type the following into Terminal: `brew install pipx`
+- Then type this: `pipx install osxphotos`
+- Now you should be able to run `osxphotos` by typing: `osxphotos`
+
+**WARNING** The git repo for this project is very large (> 1GB) because it contains multiple Photos libraries used for testing on different versions of MacOS.  If you just want to use the osxphotos package in your own code, I recommend you install the latest version from [PyPI](https://pypi.org/project/osxphotos/) which does not include all the test libraries. If you just want to use the command line utility, you can download a pre-built executable of the latest [release](https://github.com/RhetTbull/osxphotos/releases) or you can install via `pip` which also installs the command line app.  If you aren't comfortable with running python on your Mac, start with the pre-built executable or `pipx` as described above.
 
 ## Command Line Usage
 
 This package will install a command line utility called `osxphotos` that allows you to query the Photos database.  Alternatively, you can also run the command line utility like this: `python3 -m osxphotos`
 
-If you only care about the command line tool, you can download an executable of the latest [release](https://github.com/RhetTbull/osxphotos/releases).  Alternatively, I recommend installing with [pipx](https://github.com/pipxproject/pipx)
+After installing per instructions above, you should be able to run `osxphotos` on the command line:
-
-After installing pipx:
-`pipx install osxphotos`
-
-Then you should be able to run `osxphotos` on the command line:
 
 ```
 > osxphotos
@@ -114,7 +122,7 @@ Usage: osxphotos export [OPTIONS] [PHOTOS_LIBRARY]... DEST
   for photos matching all options). If no query options are provided, all
   photos will be exported. By default, all versions of all photos will be
   exported including edited versions, live photo movies, burst photos, and
-  associated RAW images.  See --skip-edited, --skip-live, --skip-bursts, and
+  associated raw images.  See --skip-edited, --skip-live, --skip-bursts, and
   --skip-raw options to modify this behavior.
 
 Options:
@@ -199,7 +207,7 @@ Options:
   --not-selfie                    Search for photos that are not selfies.
   --panorama                      Search for panorama photos.
   --not-panorama                  Search for photos that are not panoramas.
-  --has-raw                       Search for photos with both a jpeg and RAW
+  --has-raw                       Search for photos with both a jpeg and raw
                                   version
   --only-movies                   Search only for movies (default searches
                                   both images and movies).
@@ -213,6 +221,10 @@ Options:
                                   2000-01-12T12:00:00,
                                   2001-01-12T12:00:00-07:00, or 2000-12-31
                                   (ISO 8601).
+  --has-comment                   Search for photos that have comments.
+  --no-comment                    Search for photos with no comments.
+  --has-likes                     Search for photos that have likes.
+  --no-likes                      Search for photos with no likes.
   --deleted                       Include photos from the 'Recently Deleted'
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
@@ -242,10 +254,10 @@ Options:
                                   the library if a photo is a burst photo.
   --skip-live                     Do not export the associated live video
                                   component of a live photo.
-  --skip-raw                      Do not export associated RAW images of a
+  --skip-raw                      Do not export associated raw images of a
-                                  RAW/jpeg pair.  Note: this does not skip RAW
+                                  RAW+JPEG pair.  Note: this does not skip raw
-                                  photos if the RAW photo does not have an
+                                  photos if the raw photo does not have an
-                                  associated jpeg image (e.g. the RAW file was
+                                  associated jpeg image (e.g. the raw file was
                                   imported to Photos without a jpeg preview).
   --person-keyword                Use person in image as keyword/tag when
                                   exporting metadata.
@@ -279,6 +291,13 @@ Options:
                                   renamed upon import.  By default, photos are
                                   exported with the the original name they had
                                   before import.
+  --convert-to-jpeg               Convert all non-jpeg images (e.g. raw, HEIC,
+                                  PNG, etc) to JPEG upon export.  Only works
+                                  if your Mac has a GPU.
+  --jpeg-quality FLOAT RANGE      Value in range 0.0 to 1.0 to use with
+                                  --convert-to-jpeg. A value of 1.0 specifies
+                                  best quality, a value of 0.0 specifies
+                                  maximum compression. Defaults to 1.0.
   --sidecar FORMAT                Create sidecar for each photo exported;
                                   valid FORMAT values: xmp, json; --sidecar
                                   json: create JSON sidecar useable by
@@ -347,7 +366,7 @@ If using --update, the exported library should be treated as a backup, not a
 working copy where you intend to make changes.
 
 Note: The number of files reported for export and the number actually exported
-may differ due to live photos, associated RAW images, and edited photos which
+may differ due to live photos, associated raw images, and edited photos which
 are reported in the total photos exported.
 
 Implementation note: To determine which files need to be updated, osxphotos
@@ -408,23 +427,24 @@ Substitution                    Description
 {descr}                         Description of the photo
 {created.date}                  Photo's creation date in ISO format, e.g.
                                 '2020-03-22'
-{created.year}                  4-digit year of file creation time
+{created.year}                  4-digit year of photo creation time
-{created.yy}                    2-digit year of file creation time
+{created.yy}                    2-digit year of photo creation time
-{created.mm}                    2-digit month of the file creation time
+{created.mm}                    2-digit month of the photo creation time
                                 (zero padded)
-{created.month}                 Month name in user's locale of the file
+{created.month}                 Month name in user's locale of the photo
                                 creation time
 {created.mon}                   Month abbreviation in the user's locale of
-                                the file creation time
+                                the photo creation time
 {created.dd}                    2-digit day of the month (zero padded) of
-                                file creation time
+                                photo creation time
-{created.dow}                   Day of week in user's locale of the file
+{created.dow}                   Day of week in user's locale of the photo
                                 creation time
-{created.doy}                   3-digit day of year (e.g Julian day) of file
+{created.doy}                   3-digit day of year (e.g Julian day) of
-                                creation time, starting from 1 (zero padded)
+                                photo creation time, starting from 1 (zero
-{created.hour}                  2-digit hour of the file creation time
+                                padded)
-{created.min}                   2-digit minute of the file creation time
+{created.hour}                  2-digit hour of the photo creation time
-{created.sec}                   2-digit second of the file creation time
+{created.min}                   2-digit minute of the photo creation time
+{created.sec}                   2-digit second of the photo creation time
 {created.strftime}              Apply strftime template to file creation
                                 date/time. Should be used in form
                                 {created.strftime,TEMPLATE} where TEMPLATE
@@ -436,22 +456,26 @@ Substitution                    Description
                                 templates.
 {modified.date}                 Photo's modification date in ISO format,
                                 e.g. '2020-03-22'
-{modified.year}                 4-digit year of file modification time
+{modified.year}                 4-digit year of photo modification time
-{modified.yy}                   2-digit year of file modification time
+{modified.yy}                   2-digit year of photo modification time
-{modified.mm}                   2-digit month of the file modification time
+{modified.mm}                   2-digit month of the photo modification time
                                 (zero padded)
-{modified.month}                Month name in user's locale of the file
+{modified.month}                Month name in user's locale of the photo
                                 modification time
 {modified.mon}                  Month abbreviation in the user's locale of
-                                the file modification time
+                                the photo modification time
 {modified.dd}                   2-digit day of the month (zero padded) of
-                                the file modification time
+                                the photo modification time
-{modified.doy}                  3-digit day of year (e.g Julian day) of file
+{modified.dow}                  Day of week in user's locale of the photo
-                                modification time, starting from 1 (zero
+                                modification time
-                                padded)
+{modified.doy}                  3-digit day of year (e.g Julian day) of
-{modified.hour}                 2-digit hour of the file modification time
+                                photo modification time, starting from 1
-{modified.min}                  2-digit minute of the file modification time
+                                (zero padded)
-{modified.sec}                  2-digit second of the file modification time
+{modified.hour}                 2-digit hour of the photo modification time
+{modified.min}                  2-digit minute of the photo modification
+                                time
+{modified.sec}                  2-digit second of the photo modification
+                                time
 {today.date}                    Current date in iso format, e.g.
                                 '2020-03-22'
 {today.year}                    4-digit year of current date
@@ -526,6 +550,8 @@ Substitution        Description
 {label}             Image categorization label associated with a photo
                     (Photos 5 only)
 {label_normalized}  All lower case version of 'label' (Photos 5 only)
+{comment}           Comment(s) on shared Photos; format is 'Person name:
+                    comment text' (Photos 5 only)
 ```
 
 Example: export all photos to ~/Desktop/export group in folders by date created
@@ -684,7 +710,7 @@ osxphotos.PhotosDB(dbfile=path)
 
 Reads the Photos library database and returns a PhotosDB object.  
 
-Pass the path to a Photos library or to a specific database file (e.g. "/Users/smith/Pictures/Photos Library.photoslibrary" or "/Users/smith/Pictures/Photos Library.photoslibrary/database/photos.db").  Normally, it's recommended you pass the path the .photoslibrary folder, not the actual database path.  The latter option is provided for debugging -- e.g. for reading a database file if you don't have the entire library. Path to photos library may be passed **either** as first argument **or** as named argument `dbfile`. **Note**: In Photos, users may specify a different library to open by holding down the *option* key while opening Photos.app. See also [get_last_library_path](#get_last_library_path) and [get_system_library_path](#get_system_library_path)
+Pass the path to a Photos library or to a specific database file (e.g. "/Users/smith/Pictures/Photos Library.photoslibrary" or "/Users/smith/Pictures/Photos Library.photoslibrary/database/photos.db").  Normally, it's recommended you pass the path the .photoslibrary folder, not the actual database path.  **Note**: In Photos, users may specify a different library to open by holding down the *option* key while opening Photos.app. See also [get_last_library_path](#get_last_library_path) and [get_system_library_path](#get_system_library_path)
 
 If an invalid path is passed, PhotosDB will raise `FileNotFoundError` exception.
 
@@ -1084,6 +1110,18 @@ Returns the absolute path to the edited photo on disk as a string.  If the photo
 
 **Note**: will also return None if the edited photo is missing on disk. 
 
+#### `path_raw`
+Returns the absolute path to the associated raw photo on disk as a string, if photo is part of a RAW+JPEG pair, otherwise returns None.  See [notes on Raw Photos](#raw-photos).
+
+#### `has_raw`
+Returns True if photo has an associated raw image, otherwise False. (e.g. Photo is a RAW+JPEG pair). See also [is_raw](#israw) and [notes on Raw Photos](#raw-photos).
+
+#### `israw`
+Returns True if photo is a raw image. E.g. it was imported as a single raw image, not part of a RAW+JPEG pair.  See also [has_raw](#has_raw) and .
+
+#### `raw_original`
+Returns True if associated raw image and the raw image is selected in Photos via "Use RAW as Original", otherwise returns False.  See [notes on Raw Photos](#raw-photos).
+
 #### `height`
 Returns height of the photo in pixels.  If image has been edited, returns height of the edited image, otherwise returns height of the original image.  See also [original_height](#original_height).
 
@@ -1132,7 +1170,17 @@ Returns a [PlaceInfo](#PlaceInfo) object with reverse geolocation data or None i
 #### `shared`
 Returns True if photo is in a shared album, otherwise False.
 
-**Note**: *Only valid on Photos 5 / MacOS 10.15*; on Photos <= 4, returns None instead of True/False. 
+**Note**: *Only valid on Photos 5 / MacOS 10.15+; on Photos <= 4, returns None instead of True/False.
+
+#### `comments`
+Returns list of [CommentInfo](#commentinfo) objects for comments on shared photos or empty list if no comments.
+
+**Note**: *Only valid on Photos 5 / MacOS 10.15+; on Photos <= 4, returns empty list.
+
+#### `likes`
+Returns list of [LikeInfo](#likeinfo) objects for likes on shared photos or empty list if no likes.
+
+**Note**: *Only valid on Photos 5 / MacOS 10.15+; on Photos <= 4, returns empty list.
 
 #### `isphoto`
 Returns True if type is photo/still image, otherwise False
@@ -1149,7 +1197,16 @@ Returns True if photo is a [cloud asset](#iscloudasset) and is synched to iCloud
 **Note**: Applies to master (original) photo only.  It's possible for the master to be in iCloud but a local edited version is not yet synched to iCloud. `incloud` provides status of only the master photo.  osxphotos does not yet provide a means to determine if the edited version is in iCloud.  If you need this feature, please open an [issue](https://github.com/RhetTbull/osxphotos/issues).
 
 #### `uti`
-Returns Uniform Type Identifier (UTI) for the image, for example: 'public.jpeg' or 'com.apple.quicktime-movie'
+Returns Uniform Type Identifier (UTI) for the current version of the image, for example: 'public.jpeg' or 'com.apple. quicktime-movie'.  If the image has been edited, `uti` will return the UTI for the edited image, otherwise it will return the UTI for the original image.
+
+#### `uti_original`
+Returns Uniform Type Identifier (UTI) for the original unedited image, for example: 'public.jpeg' or 'com.apple.quicktime-movie'.
+
+#### `uti_edited`
+Returns Uniform Type Identifier (UTI) for the edited image, for example: 'public.jpeg'.  Returns None if the photo does not have adjustments.
+
+#### `uti_raw`
+Returns Uniform Type Identifier (UTI) for the associated raw image, if there is one; for example, 'com.canon.cr2-raw-image'.  If the image is raw but not part of a RAW+JPEG pair, `uti_raw` returns None.  In this case, use `uti`, or `uti_original`.  See also [has_raw](#has_raw) and [notes on Raw Photos](#raw-photos).
 
 #### `burst`
 Returns True if photos is a burst image (e.g. part of a set of burst images), otherwise False.
@@ -1247,7 +1304,8 @@ exiftool must be installed in the path for this to work.  If exiftool cannot be
 
 `ExifTool` provides the following methods:
 
-- `as_dict()`: returns all EXIF metadata found in the file as a dictionary in following form (Note: this shows just a subset of available metadata).  See [exiftool](https://exiftool.org/) documentation to understand which metadata keys are available.
+- `asdict()`: returns all EXIF metadata found in the file as a dictionary in following form (Note: this shows just a subset of available metadata).  See [exiftool](https://exiftool.org/) documentation to understand which metadata keys are available.
+
 ```python
 {'Composite:Aperture': 2.2,
  'Composite:GPSPosition': '-34.9188916666667 138.596861111111',
@@ -1260,7 +1318,7 @@ exiftool must be installed in the path for this to work.  If exiftool cannot be
 }
 ```
 
-- `json()`: returns same information as `as_dict()` but as a serialized JSON string.
+- `json()`: returns same information as `asdict()` but as a serialized JSON string.
 
 - `setvalue(tag, value)`: write to the EXIF data in the photo file. To delete a tag, use setvalue with value = `None`. For example:
 ```python
@@ -1271,7 +1329,7 @@ photo.exiftool.setvalue("XMP:Title", "Title of photo")
 photo.exiftool.addvalues("IPTC:Keywords", "vacation", "beach")
 ```
 
-**Caution**: I caution against writing new EXIF data to photos in the Photos library because this will overwrite the original copy of the photo and could adversely affect how Photos behaves.  `exiftool.as_dict()` is useful for getting access to all the photos information but if you want to write new EXIF data, I recommend you export the photo first then write the data.  [PhotoInfo.export()](#export) does this if called with `exiftool=True`.
+**Caution**: I caution against writing new EXIF data to photos in the Photos library because this will overwrite the original copy of the photo and could adversely affect how Photos behaves.  `exiftool.asdict()` is useful for getting access to all the photos information but if you want to write new EXIF data, I recommend you export the photo first then write the data.  [PhotoInfo.export()](#export) does this if called with `exiftool=True`.
 
 #### `score`
 Returns a [ScoreInfo](#scoreinfo) data class object which provides access to the computed aesthetic scores for each photo.
@@ -1279,7 +1337,10 @@ Returns a [ScoreInfo](#scoreinfo) data class object which provides access to the
 **Note**: Valid only for Photos 5; returns None for earlier Photos versions.
 
 #### `json()`
-Returns a JSON representation of all photo info 
+Returns a JSON representation of all photo info.
+
+#### `asdict()`
+Returns a dictionary representation of all photo info.
 
 #### `export()`
 `export(dest, *filename, edited=False, live_photo=False, export_as_hardlink=False, overwrite=False, increment=True, sidecar_json=False, sidecar_xmp=False, use_photos_export=False, timeout=120, exiftool=False, no_xattr=False, use_albums_as_keywords=False, use_persons_as_keywords=False)`
@@ -1323,21 +1384,24 @@ If overwrite=False and increment=False, export will fail if destination file alr
 
 #### <a name="rendertemplate">`render_template()`</a>
 
-`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None)`
+`render_template(template_str, none_str = "_", path_sep = None, expand_inplace = False, inplace_sep = None,         filename=False, dirname=False, replacement=":",)`
 Render template string for photo.  none_str is used if template substitution results in None value and no default specified. 
 - `template_str`: str in form "{name,DEFAULT}" where name is one of the values in table below. The "," and default value that follows are optional. If specified, "DEFAULT" will be used if "name" is None.  This is useful for values which are not always present, for example reverse geolocation data.
 - `none_str`: optional str to use as substitution when template value is None and no default specified in the template string.  default is "_".
 - `path_sep`: optional character to use as path separator, default is os.path.sep
 - `expand_inplace`: expand multi-valued substitutions in-place as a single string instead of returning individual strings
 - `inplace_sep`: optional string to use as separator between multi-valued keywords with expand_inplace; default is ','
+- `filename`: if True, template output will be sanitized to produce valid file name
+- `dirname`: if True, template output will be sanitized to produce valid directory name
+- `replacement`: str, value to replace any illegal file path characters with; default = ":"
 
 Returns a tuple of (rendered, unmatched) where rendered is a list of rendered strings with all substitutions made and unmatched is a list of any strings that resembled a template substitution but did not match a known substitution. E.g. if template contained "{foo}", unmatched would be ["foo"].
 
-e.g. `render_filepath_template("{created.year}/{foo}", photo)` would return `(["2020/{foo}"],["foo"])`
+e.g. `render_template("{created.year}/{foo}", photo)` would return `(["2020/{foo}"],["foo"])`
 
 If you want to include "{" or "}" in the output, use "{{" or "}}"
 
-e.g. `render_filepath_template("{created.year}/{{foo}}", photo)` would return `(["2020/{foo}"],[])`
+e.g. `render_template("{created.year}/{{foo}}", photo)` would return `(["2020/{foo}"],[])`
 
 Some substitutions, notably `album`, `keyword`, and `person` could return multiple values, hence a new string will be return for each possible substitution (hence why a list of rendered strings is returned).  For example, a photo in 2 albums: 'Vacation' and 'Family' would result in the following rendered values if template was "{created.year}/{album}" and created.year == 2020: `["2020/Vacation","2020/Family"]` 
 
@@ -1458,6 +1522,11 @@ Returns the title or name of the folder.
 #### `album_info`
 Returns a list of [AlbumInfo](#AlbumInfo) objects representing each album contained in the folder.
 
+#### `album_info_shared`
+Returns a list of [AlbumInfo](#AlbumInfo) objects for each shared album in the photos database.
+
+**Note**: Only valid for Photos 5+; on Photos <= 4, prints warning and returns empty list.
+
 #### `subfolders`
 Returns a list of [FolderInfo](#FolderInfo) objects representing the sub-folders of the folder.  
 
@@ -1619,6 +1688,9 @@ Returns a list of [FaceInfo](#faceinfo) objects associated with this person sort
 #### `json()`
 Returns a json string representation of the PersonInfo instance.
 
+#### `asdict()`
+Returns a dictionary representation of the PersonInfo instance.
+
 ### FaceInfo 
 [PhotoInfo.face_info](#photofaceinfo) return a list of FaceInfo objects representing detected faces in a photo.  The FaceInfo class has the following properties and methods.
 
@@ -1704,6 +1776,53 @@ Returns a dictionary representation of the FaceInfo instance.
 #### `json()`
 Returns a JSON representation of the FaceInfo instance.
 
+### CommentInfo
+[PhotoInfo.comments](#comments) returns a list of CommentInfo objects for comments on shared photos. (Photos 5/MacOS 10.15+ only).  The list of CommentInfo objects will be sorted in ascending order by date comment was made.  CommentInfo contains the following fields:
+
+- `datetime`: `datetime.datetime`, date/time comment was made
+- `user`: `str`, name of user who made the comment
+- `ismine`: `bool`, True if comment was made by person who owns the Photos library being operated on
+- `text`: `str`, text of the actual comment
+
+### LikeInfo
+[PhotoInfo.likes](#likes) returns a list of LikeInfo objects for "likes" on shared photos. (Photos 5/MacOS 10.15+ only).  The list of LikeInfo objects will be sorted in ascending order by date like was made.  LikeInfo contains the following fields:
+
+- `datetime`: `datetime.datetime`, date/time like was made
+- `user`: `str`, name of user who made the like 
+- `ismine`: `bool`, True if like was made by person who owns the Photos library being operated on
+
+### Raw Photos
+Handling raw photos in `osxphotos` requires a bit of extra work.  Raw photos in Photos can be imported in two different ways: 1) a single raw photo with no associated JPEG image is imported 2) a raw+JPEG pair is imported -- two separate images with same file stem (e.g. `IMG_0001.CR2` and `IMG_001.JPG`) are imported.  
+
+The latter are treated by Photos as a single image.  By default, Photos will treat these as a JPEG image.  They are denoted in the Photos interface with a "J" icon superimposed on the image.  In Photos, the user can select "Use RAW as original" in which case the "J" icon changes to an "R" icon and all subsequent edits will use the raw image as the original. To further complicate this, different versions of Photos handle these differently in their internal logic.  
+
+`osxphotos` attempts to simplify the handling of these raw+JPEG pairs by providing a set of attributes for accessing both the JPEG and the raw version.  For example, [PhotoInfo.has_raw](#has_raw) will be True if the photo has an associated raw image but False otherwise and [PhotoInfo.path_raw](#path_raw) provides the path to the associated raw image.  Reference the following table for the various attributes useful for dealing with raw images.  Given the different ways Photos deals with raw images I've struggled with how to represent these in a logical and consistent manner.  If you have suggestions for a better interface, please open an [issue](https://github.com/RhetTbull/osxphotos/issues)!
+
+#### Raw-Related Attributes 
+
+|`PhotoInfo` attribute|`IMG_0001.CR2` imported without raw+JPEG pair|`IMG_0001.CR2` + `IMG_0001.JPG` raw+JPEG pair, JPEG is original|`IMG_0001.CR2` + `IMG_0001.JPG` raw+jpeg pair, raw is original|
+|----------|----------|----------|----------|
+|[israw](#israw)| True | False | False |
+|[has_raw](#has_raw)| False | True | True |
+|[uti](#uti) | `com.canon.cr2-raw-image` | `public.jpeg` | `public.jpeg` |
+|[uti_raw](#uti_raw) | None | `com.canon.cr2-raw-image` | `com.canon.cr2-raw-image` |
+|[raw_original](#raw_original) | False | False | True |
+|[path](#path) | `/path/to/IMG_0001.CR2` | `/path/to/IMG_0001.JPG` | `/path/to/IMG_0001.JPG` |
+|[path_raw](#path_raw) | None | `/path/to/IMG_0001.CR2` | `/path/to/IMG_0001.CR2` | 
+
+#### Example
+To get the path of every raw photo, whether it's a single raw photo or a raw+JPEG pair, one could do something like this:
+
+```python
+>>> import osxphotos
+>>> photosdb = osxphotos.PhotosDB()
+>>> photos = photosdb.photos()
+>>> all_raw = [p for p in photos if p.israw or p.has_raw]
+>>> for raw in all_raw:
+...     path = raw.path if raw.israw else raw.path_raw
+...     print(path)
+```
+
 ### Template Substitutions
 
 The following substitutions are availabe for use with `PhotoInfo.render_template()` 
@@ -1769,6 +1888,7 @@ The following substitutions are availabe for use with `PhotoInfo.render_template
 |{person}|Person(s) / face(s) in a photo|
 |{label}|Image categorization label associated with a photo (Photos 5 only)|
 |{label_normalized}|All lower case version of 'label' (Photos 5 only)|
+|{comment}|Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)|
 
 ### Utility Functions
 
@@ -1851,6 +1971,7 @@ if __name__ == "__main__":
 - [rhettbull/photosmeta](https://github.com/rhettbull/photosmeta): uses osxphotos and [exiftool](https://exiftool.org/) to apply metadata from Photos as exif data in the photo files.  Can also export photos while preserving metadata and also apply Photos keywords as spotlight tags to make it easier to search for photos using spotlight.  This is mostly made obsolete by osxphotos.  The one feature that photosmeta has that osxphotos does not is ability to update the metadata of the actual photo files in the Photos library without exporting them. (Use with caution!)
 - [rhettbull/PhotoScript](https://github.com/RhetTbull/PhotoScript): python wrapper around Photos' applescript API allowing automation of Photos (including creation/deletion of items) from python.
 - [patrikhson/photo-export](https://github.com/patrikhson/photo-export): Exports older versions of Photos databases.  Provided the inspiration for osxphotos.
+- [doersino/apple-photos-export](https://github.com/doersino/apple-photos-export): Photos export script for Mojave.
 - [orangeturtle739/photos-export](https://github.com/orangeturtle739/photos-export): Set of scripts to export Photos libraries.
 - [ndbroadbent/icloud_photos_downloader](https://github.com/ndbroadbent/icloud_photos_downloader): Download photos from iCloud.  Currently unmaintained.
 - [AaronVanGeffen/ExportPhotosLibrary](https://github.com/AaronVanGeffen/ExportPhotosLibrary): Another python script for exporting older versions of Photos libraries.
@@ -1885,10 +2006,10 @@ Thank-you to the following people who have contributed to improving osxphotos!
 
 ## Known Bugs
 
-My goal is make osxphotos as reliable and comprehensive as possible.  The test suite currently has over 600 tests--but there are still some [bugs](https://github.com/RhetTbull/osxphotos/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or incomplete features lurking.  If you find bugs please open an [issue](https://github.com/RhetTbull/osxphotos/issues).  Notable issues include:
+My goal is make osxphotos as reliable and comprehensive as possible.  The test suite currently has over 800 tests--but there are still some [bugs](https://github.com/RhetTbull/osxphotos/issues?q=is%3Aissue+is%3Aopen+label%3Abug) or incomplete features lurking.  If you find bugs please open an [issue](https://github.com/RhetTbull/osxphotos/issues).  Please consult the list of open bugs before deciding that you want to use this code on your Photos library.  Notable issues include:
 
 - Face coordinates (mouth, left eye, right eye) may not be correct for images where the head is tilted.  See [Issue #196](https://github.com/RhetTbull/osxphotos/issues/196).
-- RAW images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the RAW image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the RAW image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
+- Raw images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the raw image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the raw image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75).
 
 ## Implementation Notes
@@ -1908,6 +2029,8 @@ For additional details about how osxphotos is implemented or if you would like t
 - [Mako](https://www.makotemplates.org/)
 - [bpylist2](https://pypi.org/project/bpylist2/)
 - [pathvalidate](https://pypi.org/project/pathvalidate/)
+- [wurlitzer](https://pypi.org/project/wurlitzer/)
+
 
 ## Acknowledgements
 This project was originally inspired by [photo-export](https://github.com/patrikhson/photo-export) by Patrick Fältström,  Copyright (c) 2015 Patrik Fältström paf@frobbit.se

examples/get_shared_photo_comments.py

@@ -0,0 +1,156 @@
+""" get shared comments associated with a photo """
+
+import datetime
+import sys
+from dataclasses import dataclass
+
+import osxphotos
+from osxphotos._constants import TIME_DELTA
+
+
+@dataclass
+class Comment:
+    """ Class for shared photo comments """
+
+    uuid: str
+    sort_fok: int
+    datetime: datetime.datetime
+    user: str
+    ismine: bool
+    text: str
+
+
+@dataclass
+class Like:
+    """ Class for shared photo likes """
+
+    uuid: str
+    sort_fok: int
+    datetime: datetime.datetime
+    user: str
+    ismine: bool
+
+
+def get_shared_person_info(photosdb, hashed_person_id):
+    """ returns tuple of (first name, last name, full name) 
+        for person invited to shared album with 
+        ZINVITEEHASHEDPERSONID = hashed_person_id
+        
+    Args:
+        photosdb: a osxphotos.PhotosDB object
+        hashed_person_id: str, value of ZINVITEEHASHEDPERSONID to lookup
+    """
+
+    conn, _ = photosdb.get_db_connection()
+    results = conn.execute(
+        """
+        SELECT
+        ZINVITEEHASHEDPERSONID,
+        ZINVITEEFIRSTNAME,
+        ZINVITEELASTNAME,
+        ZINVITEEFULLNAME
+        FROM
+        ZCLOUDSHAREDALBUMINVITATIONRECORD
+        WHERE
+        ZINVITEEHASHEDPERSONID = ?
+        LIMIT 1
+        """,
+        ([hashed_person_id]),
+    ).fetchall()
+
+    if results:
+        row = results[0]
+        return (row[1], row[2], row[3])
+    else:
+        return (None, None, None)
+
+
+def get_comments(photosdb, uuid):
+    """ return comments and likes, if any, for photo with uuid
+
+    Args:
+        photosdb: a osxphotos.PhotosDB object
+        uuid: uuid of the photo
+    
+    Returns:
+        tuple of (list of comments as Comment objects or [] if no comments, list of likes as Like objects or [] if no likes)
+    """
+    conn, _ = photosdb.get_db_connection()
+
+    results = conn.execute(
+        """
+        SELECT 
+        ZGENERICASSET.ZUUID, --0: UUID of the photo
+        ZCLOUDSHAREDCOMMENT.ZISLIKE, --1: comment is actually a "like"
+        ZCLOUDSHAREDCOMMENT.Z_FOK_COMMENTEDASSET, --2: sort order for comments on a photo
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTDATE, --3: date of comment
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTTEXT, --4: text of comment
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTERHASHEDPERSONID, --5: hashed ID of person who made comment/like
+        ZCLOUDSHAREDCOMMENT.ZISMYCOMMENT --6: is my (this user's) comment
+        FROM ZCLOUDSHAREDCOMMENT
+        JOIN ZGENERICASSET ON
+        ZGENERICASSET.Z_PK = ZCLOUDSHAREDCOMMENT.ZCOMMENTEDASSET
+        OR
+        ZGENERICASSET.Z_PK = ZCLOUDSHAREDCOMMENT.ZLIKEDASSET
+        WHERE ZGENERICASSET.ZUUID = ?
+    """,
+        ([uuid]),
+    ).fetchall()
+
+    comments = []
+    likes = []
+    for row in results:
+        photo_uuid = row[0]
+        sort_fok = row[2] or 0  # sort_fok is Null/None for likes
+        is_like = bool(row[1])
+        text = row[4]
+        user_info = get_shared_person_info(photosdb, row[5])
+        try:
+            dt = datetime.datetime.fromtimestamp(row[3] + TIME_DELTA)
+        except:
+            dt = datetime.datetime(1970, 1, 1)
+        ismine = bool(row[6])
+        if is_like:
+            # it's a like
+            likes.append(Like(photo_uuid, sort_fok, dt, user_info[2], ismine))
+        elif text:
+            # comment
+            comments.append(
+                Comment(photo_uuid, sort_fok, dt, user_info[2], ismine, text)
+            )
+    if likes:
+        likes.sort(key=lambda x: x.datetime)
+    if comments:
+        comments.sort(key=lambda x: x.sort_fok)
+    return (comments, likes)
+
+
+def main():
+    if len(sys.argv) > 1:
+        # library as first argument
+        photosdb = osxphotos.PhotosDB(dbfile=sys.argv[1])
+    else:
+        # open default library
+        photosdb = osxphotos.PhotosDB()
+
+    # shared albums
+    shared_albums = photosdb.album_info_shared
+    for album in shared_albums:
+        print(f"Processing album {album.title}")
+        # only shared albums can have comments
+        for photo in album.photos:
+            comments, likes = get_comments(photosdb, photo.uuid)
+            if comments or likes:
+                print(f"{photo.uuid}, {photo.original_filename}: ")
+            if likes:
+                print("Likes:")
+                for like in likes:
+                    print(like)
+            if comments:
+                print("Comments:")
+                for comment in comments:
+                    print(comment)
+
+
+if __name__ == "__main__":
+    main()

examples/photos_repl.py

@@ -42,7 +42,7 @@ def main():
     if db:
         print("loading database")
         tic = time.perf_counter()
-        photosdb = osxphotos.PhotosDB(dbfile=db)
+        photosdb = osxphotos.PhotosDB(dbfile=db, verbose=print)
         toc = time.perf_counter()
         print(f"done: took {toc-tic} seconds")
         return photosdb

osxphotos/__init__.py

@@ -1,8 +1,7 @@
-import logging
-
 from ._version import __version__
 from .photoinfo import PhotoInfo
 from .photosdb import PhotosDB
+from .photosdb._photosdb_process_comments import CommentInfo, LikeInfo
 from .phototemplate import PhotoTemplate
 from .utils import _debug, _get_logger, _set_debug
 

osxphotos/__main__.py

@@ -1,9 +1,7 @@
 """ command line interface for osxphotos """
 import csv
 import datetime
-import functools
 import json
-import logging
 import os
 import os.path
 import pathlib
@@ -14,12 +12,6 @@ import unicodedata
 
 import click
 import yaml
-from pathvalidate import (
-    is_valid_filename,
-    is_valid_filepath,
-    sanitize_filename,
-    sanitize_filepath,
-)
 
 import osxphotos
 
@@ -29,11 +21,12 @@ from ._constants import (
     _UNKNOWN_PLACE,
     UNICODE_FORMAT,
 )
-from ._export_db import ExportDB, ExportDBInMemory
 from ._version import __version__
 from .datetime_formatter import DateTimeFormatter
 from .exiftool import get_exiftool_path
+from .export_db import ExportDB, ExportDBInMemory
 from .fileutil import FileUtil, FileUtilNoOp
+from .path_utils import is_valid_filepath, sanitize_filename, sanitize_filepath
 from .photoinfo import ExportResults
 from .phototemplate import TEMPLATE_SUBSTITUTIONS, TEMPLATE_SUBSTITUTIONS_MULTI_VALUED
 
@@ -148,7 +141,7 @@ class ExportCommand(click.Command):
         formatter.write("\n")
         formatter.write_text(
             "Note: The number of files reported for export and the number actually exported "
-            + "may differ due to live photos, associated RAW images, and edited photos which are reported "
+            + "may differ due to live photos, associated raw images, and edited photos which are reported "
             + "in the total photos exported."
         )
         formatter.write("\n")
@@ -474,7 +467,7 @@ def query_options(f):
         o(
             "--has-raw",
             is_flag=True,
-            help="Search for photos with both a jpeg and RAW version",
+            help="Search for photos with both a jpeg and raw version",
         ),
         o(
             "--only-movies",
@@ -496,6 +489,10 @@ def query_options(f):
             help="Search by end item date, e.g. 2000-01-12T12:00:00, 2001-01-12T12:00:00-07:00, or 2000-12-31 (ISO 8601).",
             type=DateTimeISO8601(),
         ),
+        o("--has-comment", is_flag=True, help="Search for photos that have comments."),
+        o("--no-comment", is_flag=True, help="Search for photos with no comments."),
+        o("--has-likes", is_flag=True, help="Search for photos that have likes."),
+        o("--no-likes", is_flag=True, help="Search for photos with no likes."),
     ]
     for o in options[::-1]:
         f = o(f)
@@ -528,10 +525,15 @@ def cli(ctx, db, json_, debug):
     help="Use with '--dump photos' to dump only certain UUIDs",
     multiple=True,
 )
+@click.option("--verbose", "-V", "verbose_", is_flag=True, help="Print verbose output.")
 @click.pass_obj
 @click.pass_context
-def debug_dump(ctx, cli_obj, db, photos_library, dump, uuid):
+def debug_dump(ctx, cli_obj, db, photos_library, dump, uuid, verbose_):
     """ Print out debug info """
+
+    global VERBOSE
+    VERBOSE = bool(verbose_)
+
     db = get_photos_db(*photos_library, db, cli_obj.db)
     if db is None:
         click.echo(cli.commands["debug-dump"].get_help(ctx), err=True)
@@ -541,7 +543,7 @@ def debug_dump(ctx, cli_obj, db, photos_library, dump, uuid):
 
     start_t = time.perf_counter()
     print(f"Opening database: {db}")
-    photosdb = osxphotos.PhotosDB(dbfile=db)
+    photosdb = osxphotos.PhotosDB(dbfile=db, verbose=verbose)
     stop_t = time.perf_counter()
     print(f"Done; took {(stop_t-start_t):.2f} seconds")
 
@@ -984,6 +986,10 @@ def query(
     label,
     deleted,
     deleted_only,
+    has_comment,
+    no_comment,
+    has_likes,
+    no_likes,
 ):
     """ Query the Photos database using 1 or more search options; 
         if more than one option is provided, they are treated as "AND" 
@@ -1027,6 +1033,9 @@ def query(
         (panorama, not_panorama),
         (any(place), no_place),
         (deleted, deleted_only),
+        (shared, not_shared),
+        (has_comment, no_comment),
+        (has_likes, no_likes),
     ]
     # print help if no non-exclusive term or a double exclusive term is given
     if any(all(bb) for bb in exclusive) or not any(
@@ -1113,6 +1122,10 @@ def query(
         label=label,
         deleted=deleted,
         deleted_only=deleted_only,
+        has_comment=has_comment,
+        no_comment=no_comment,
+        has_likes=has_likes,
+        no_likes=no_likes,
     )
 
     # below needed for to make CliRunner work for testing
@@ -1183,9 +1196,9 @@ def query(
 @click.option(
     "--skip-raw",
     is_flag=True,
-    help="Do not export associated RAW images of a RAW/jpeg pair.  "
+    help="Do not export associated raw images of a RAW+JPEG pair.  "
-    "Note: this does not skip RAW photos if the RAW photo does not have an associated jpeg image "
+    "Note: this does not skip raw photos if the raw photo does not have an associated jpeg image "
-    "(e.g. the RAW file was imported to Photos without a jpeg preview).",
+    "(e.g. the raw file was imported to Photos without a jpeg preview).",
 )
 @click.option(
     "--person-keyword",
@@ -1230,6 +1243,21 @@ def query(
     "Note: Starting with Photos 5, all photos are renamed upon import.  By default, "
     "photos are exported with the the original name they had before import.",
 )
+@click.option(
+    "--convert-to-jpeg",
+    is_flag=True,
+    help="Convert all non-jpeg images (e.g. raw, HEIC, PNG, etc) "
+    "to JPEG upon export.  Only works if your Mac has a GPU.",
+)
+@click.option(
+    "--jpeg-quality",
+    type=click.FloatRange(0.0, 1.0),
+    default=1.0,
+    help="Value in range 0.0 to 1.0 to use with --convert-to-jpeg. "
+    "A value of 1.0 specifies best quality, "
+    "a value of 0.0 specifies maximum compression. "
+    "Defaults to 1.0.",
+)
 @click.option(
     "--sidecar",
     default=None,
@@ -1349,6 +1377,8 @@ def export(
     keyword_template,
     description_template,
     current_name,
+    convert_to_jpeg,
+    jpeg_quality,
     sidecar,
     only_photos,
     only_movies,
@@ -1379,6 +1409,10 @@ def export(
     edited_suffix,
     place,
     no_place,
+    has_comment,
+    no_comment,
+    has_likes,
+    no_likes,
     no_extended_attributes,
     label,
     deleted,
@@ -1392,13 +1426,13 @@ def export(
         (e.g. search for photos matching all options).
         If no query options are provided, all photos will be exported.
         By default, all versions of all photos will be exported including edited
-        versions, live photo movies, burst photos, and associated RAW images. 
+        versions, live photo movies, burst photos, and associated raw images. 
         See --skip-edited, --skip-live, --skip-bursts, and --skip-raw options
         to modify this behavior. 
     """
 
     global VERBOSE
-    VERBOSE = True if verbose_ else False
+    VERBOSE = bool(verbose_)
 
     if not os.path.isdir(dest):
         sys.exit(f"DEST {dest} must be valid path")
@@ -1424,6 +1458,10 @@ def export(
         (any(place), no_place),
         (deleted, deleted_only),
         (skip_edited, skip_original_if_edited),
+        (export_as_hardlink, convert_to_jpeg),
+        (shared, not_shared),
+        (has_comment, no_comment),
+        (has_likes, no_likes),
     ]
     if any(all(bb) for bb in exclusive):
         click.echo("Incompatible export options", err=True)
@@ -1491,13 +1529,22 @@ def export(
 
     if dry_run:
         export_db = ExportDBInMemory(export_db_path)
-        # echo = functools.partial(click.echo, err=True)
-        # fileutil = FileUtilNoOp(verbose=echo)
         fileutil = FileUtilNoOp
     else:
         export_db = ExportDB(export_db_path)
         fileutil = FileUtil
 
+    if verbose_:
+        if export_db.was_created:
+            verbose(f"Created export database {export_db_path}")
+        else:
+            verbose(f"Using export database {export_db_path}")
+        upgraded = export_db.was_upgraded
+        if upgraded:
+            verbose(
+                f"Upgraded export database {export_db_path} from version {upgraded[0]} to {upgraded[1]}"
+            )
+
     photos = _query(
         db=db,
         keyword=keyword,
@@ -1553,6 +1600,10 @@ def export(
         label=label,
         deleted=deleted,
         deleted_only=deleted_only,
+        has_comment=has_comment,
+        no_comment=no_comment,
+        has_likes=has_likes,
+        no_likes=no_likes,
     )
 
     if photos:
@@ -1610,6 +1661,8 @@ def export(
                     touch_file=touch_file,
                     edited_suffix=edited_suffix,
                     use_photos_export=use_photos_export,
+                    convert_to_jpeg=convert_to_jpeg,
+                    jpeg_quality=jpeg_quality,
                 )
                 results_exported.extend(results.exported)
                 results_new.extend(results.new)
@@ -1618,6 +1671,12 @@ def export(
                 results_exif_updated.extend(results.exif_updated)
                 results_touched.extend(results.touched)
 
+                # if convert_to_jpeg and p.isphoto and p.uti != "public.jpeg":
+                #     for photo_file in set(
+                #         results.exported + results.updated + results.exif_updated
+                #     ):
+                #         verbose(f"Converting {photo_file} to jpeg")
+
         else:
             # show progress bar
             with click.progressbar(photos) as bar:
@@ -1651,6 +1710,8 @@ def export(
                         touch_file=touch_file,
                         edited_suffix=edited_suffix,
                         use_photos_export=use_photos_export,
+                        convert_to_jpeg=convert_to_jpeg,
+                        jpeg_quality=jpeg_quality,
                     )
                     results_exported.extend(results.exported)
                     results_new.extend(results.new)
@@ -1658,6 +1719,7 @@ def export(
                     results_skipped.extend(results.skipped)
                     results_exif_updated.extend(results.exif_updated)
                     results_touched.extend(results.touched)
+
         stop_time = time.perf_counter()
         # print summary results
         if update:
@@ -1861,13 +1923,17 @@ def _query(
     label=None,
     deleted=False,
     deleted_only=False,
+    has_comment=False,
+    no_comment=False,
+    has_likes=False,
+    no_likes=False,
 ):
     """ run a query against PhotosDB to extract the photos based on user supply criteria 
         used by query and export commands 
         arguments must be passed in same order as query and export 
         if either is modified, need to ensure all three functions are updated """
 
-    photosdb = osxphotos.PhotosDB(dbfile=db)
+    photosdb = osxphotos.PhotosDB(dbfile=db, verbose=verbose)
     if deleted or deleted_only:
         photos = photosdb.photos(
             uuid=uuid,
@@ -2020,7 +2086,7 @@ def _query(
         photos = [p for p in photos if not p.shared]
 
     if uti:
-        photos = [p for p in photos if uti in p.uti]
+        photos = [p for p in photos if uti in p.uti_original]
 
     if burst:
         photos = [p for p in photos if p.burst]
@@ -2080,6 +2146,16 @@ def _query(
     if has_raw:
         photos = [p for p in photos if p.has_raw]
 
+    if has_comment:
+        photos = [p for p in photos if p.comments]
+    elif no_comment:
+        photos = [p for p in photos if not p.comments]
+
+    if has_likes:
+        photos = [p for p in photos if p.likes]
+    elif no_likes:
+        photos = [p for p in photos if not p.likes]
+
     return photos
 
 
@@ -2140,6 +2216,8 @@ def export_photo(
     touch_file=None,
     edited_suffix="_edited",
     use_photos_export=False,
+    convert_to_jpeg=False,
+    jpeg_quality=1.0,
 ):
     """ Helper function for export that does the actual export
 
@@ -2159,7 +2237,7 @@ def export_photo(
         directory: template used to determine output directory
         filename_template: template use to determine output file
         no_extended_attributes: boolean; if True, exports photo without preserving extended attributes
-        export_raw: boolean; if True exports RAW image associate with the photo
+        export_raw: boolean; if True exports raw image associate with the photo
         export_edited: boolean; if True exports edited version of photo if there is one
         skip_original_if_edited: boolean; if True does not export original if photo has been edited
         album_keyword: boolean; if True, exports album names as keywords in metadata
@@ -2171,6 +2249,8 @@ def export_photo(
         dry_run: boolean; if True, doesn't actually export or update any files
         touch_file: boolean; sets file's modification time to match photo date
         use_photos_export: boolean; if True forces the use of AppleScript to export even if photo not missing
+        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
     Returns:
         list of path(s) of exported photo or None if photo was missing
@@ -2179,7 +2259,7 @@ def export_photo(
         ValueError on invalid filename_template
     """
     global VERBOSE
-    VERBOSE = True if verbose_ else False
+    VERBOSE = bool(verbose_)
 
     if not download_missing:
         if photo.ismissing:
@@ -2193,7 +2273,7 @@ def export_photo(
                 f"skipping {photo.original_filename}"
             )
             return ExportResults([], [], [], [], [], [])
-    elif photo.ismissing and not photo.iscloudasset or not photo.incloud:
+    elif photo.ismissing and not photo.iscloudasset and not photo.incloud:
         verbose(
             f"Skipping missing {photo.original_filename}: not iCloud asset or missing from cloud"
         )
@@ -2257,6 +2337,8 @@ def export_photo(
                     fileutil=fileutil,
                     dry_run=dry_run,
                     touch_file=touch_file,
+                    convert_to_jpeg=convert_to_jpeg,
+                    jpeg_quality=jpeg_quality,
                 )
 
                 results_exported.extend(export_results.exported)
@@ -2316,6 +2398,8 @@ def export_photo(
                         fileutil=fileutil,
                         dry_run=dry_run,
                         touch_file=touch_file,
+                        convert_to_jpeg=convert_to_jpeg,
+                        jpeg_quality=jpeg_quality,
                     )
 
                     results_exported.extend(export_results_edited.exported)
@@ -2363,7 +2447,9 @@ def get_filenames_from_template(photo, filename_template, original_name):
     """
     if filename_template:
         photo_ext = pathlib.Path(photo.original_filename).suffix
-        filenames, unmatched = photo.render_template(filename_template, path_sep="_")
+        filenames, unmatched = photo.render_template(
+            filename_template, path_sep="_", filename=True
+        )
         if not filenames or unmatched:
             raise click.BadOptionUsage(
                 "filename_template",
@@ -2372,6 +2458,8 @@ def get_filenames_from_template(photo, filename_template, original_name):
         filenames = [f"{file_}{photo_ext}" for file_ in filenames]
     else:
         filenames = [photo.original_filename] if original_name else [photo.filename]
+
+    filenames = [sanitize_filename(filename) for filename in filenames]
     return filenames
 
 
@@ -2402,22 +2490,18 @@ def get_dirnames_from_template(photo, directory, export_by_date, dest, dry_run):
         dest_paths = [dest_path]
     elif directory:
         # got a directory template, render it and check results are valid
-        dirnames, unmatched = photo.render_template(directory)
+        dirnames, unmatched = photo.render_template(directory, dirname=True)
-        if not dirnames:
+        if not dirnames or unmatched:
-            raise click.BadOptionUsage(
-                "directory",
-                f"Invalid template '{directory}': results={dirnames} unmatched={unmatched}",
-            )
-        elif unmatched:
             raise click.BadOptionUsage(
                 "directory",
                 f"Invalid template '{directory}': results={dirnames} unmatched={unmatched}",
             )
+
         dest_paths = []
         for dirname in dirnames:
-            dirname = sanitize_filepath(dirname, platform="auto")
+            dirname = sanitize_filepath(dirname)
             dest_path = os.path.join(dest, dirname)
-            if not is_valid_filepath(dest_path, platform="auto"):
+            if not is_valid_filepath(dest_path):
                 raise ValueError(f"Invalid file path: '{dest_path}'")
             if not dry_run and not os.path.isdir(dest_path):
                 os.makedirs(dest_path)
@@ -2445,7 +2529,7 @@ def find_files_in_branch(pathname, filename):
     files = []
 
     # walk down the tree
-    for root, directories, filenames in os.walk(pathname):
+    for root, _, filenames in os.walk(pathname):
         # for directory in directories:
         # print(os.path.join(root, directory))
         for fname in filenames:

osxphotos/_constants.py

@@ -102,3 +102,10 @@ _OSXPHOTOS_NONE_SENTINEL = "OSXPhotosXYZZY42_Sentinel$"
 
 # SearchInfo categories for Photos 5, corresponds to categories in database/search/psi.sqlite
 SEARCH_CATEGORY_LABEL = 2024
+
+# Max filename length on MacOS
+MAX_FILENAME_LEN = 255
+
+# Max directory name length on MacOS
+MAX_DIRNAME_LEN = 255
+

osxphotos/_version.py

@@ -1,3 +1,4 @@
 """ version info """
 
-__version__ = "0.34.5"
+__version__ = "0.36.2"
+

osxphotos/exiftool.py

@@ -228,7 +228,7 @@ class ExifTool:
         ver = self.run_commands("-ver", no_file=True)
         return ver.decode("utf-8")
 
-    def as_dict(self):
+    def asdict(self):
         """ return dictionary of all EXIF tags and values from exiftool 
             returns empty dict if no tags
         """
@@ -245,7 +245,7 @@ class ExifTool:
 
     def _read_exif(self):
         """ read exif data from file """
-        data = self.as_dict()
+        data = self.asdict()
         self.data = {k: v for k, v in data.items()}
 
     def __str__(self):

osxphotos/export_db.py

@@ -14,7 +14,7 @@ from sqlite3 import Error
 
 from ._version import __version__
 
-OSXPHOTOS_EXPORTDB_VERSION = "1.0"
+OSXPHOTOS_EXPORTDB_VERSION = "2.0"
 
 
 class ExportDB_ABC(ABC):
@@ -36,6 +36,22 @@ class ExportDB_ABC(ABC):
     def get_stat_orig_for_file(self, filename):
         pass
 
+    @abstractmethod
+    def set_stat_edited_for_file(self, filename, stats):
+        pass
+
+    @abstractmethod
+    def get_stat_edited_for_file(self, filename):
+        pass
+
+    @abstractmethod
+    def set_stat_converted_for_file(self, filename, stats):
+        pass
+
+    @abstractmethod
+    def get_stat_converted_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_stat_exif_for_file(self, filename, stats):
         pass
@@ -61,13 +77,28 @@ class ExportDB_ABC(ABC):
         pass
 
     @abstractmethod
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         pass
 
 
 class ExportDBNoOp(ExportDB_ABC):
     """ An ExportDB with NoOp methods """
 
+    def __init__(self):
+        self.was_created = True
+        self.was_upgraded = False
+        self.version = OSXPHOTOS_EXPORTDB_VERSION
+
     def get_uuid_for_file(self, filename):
         pass
 
@@ -80,6 +111,18 @@ class ExportDBNoOp(ExportDB_ABC):
     def get_stat_orig_for_file(self, filename):
         pass
 
+    def set_stat_edited_for_file(self, filename, stats):
+        pass
+
+    def get_stat_edited_for_file(self, filename):
+        pass
+
+    def set_stat_converted_for_file(self, filename, stats):
+        pass
+
+    def get_stat_converted_for_file(self, filename):
+        pass
+
     def set_stat_exif_for_file(self, filename, stats):
         pass
 
@@ -98,7 +141,17 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_exifdata_for_file(self, uuid, exifdata):
         pass
 
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         pass
 
 
@@ -122,7 +175,6 @@ class ExportDB(ExportDB_ABC):
             returns None if filename not found in database
         """
         filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
-        logging.debug(f"get_uuid: {filename}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -135,14 +187,12 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             uuid = None
 
-        logging.debug(f"get_uuid: {uuid}")
         return uuid
 
     def set_uuid_for_file(self, filename, uuid):
         """ set UUID of filename to uuid in the database """
         filename = str(pathlib.Path(filename).relative_to(self._path))
         filename_normalized = filename.lower()
-        logging.debug(f"set_uuid: {filename} {uuid}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -162,7 +212,6 @@ class ExportDB(ExportDB_ABC):
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
-        logging.debug(f"set_stat_orig_for_file: {filename} {stats}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -199,9 +248,20 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             stats = (None, None, None)
 
-        logging.debug(f"get_stat_orig_for_file: {stats}")
         return stats
 
+    def set_stat_edited_for_file(self, filename, stats):
+        """ set stat info for edited version of image (in Photos' library)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._set_stat_for_file("edited", filename, stats)
+
+    def get_stat_edited_for_file(self, filename):
+        """ get stat info for edited version of image (in Photos' library)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._get_stat_for_file("edited", filename)
+
     def set_stat_exif_for_file(self, filename, stats):
         """ set stat info for filename (after exiftool has updated it)
             filename: filename to set the stat info for
@@ -210,7 +270,6 @@ class ExportDB(ExportDB_ABC):
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
-        logging.debug(f"set_stat_exif_for_file: {filename} {stats}")
         conn = self._conn
         try:
             c = conn.cursor()
@@ -247,9 +306,20 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             stats = (None, None, None)
 
-        logging.debug(f"get_stat_exif_for_file: {stats}")
         return stats
 
+    def set_stat_converted_for_file(self, filename, stats):
+        """ set stat info for filename (after image converted to jpeg)
+            filename: filename to set the stat info for
+            stat: a tuple of length 3: mode, size, mtime """
+        return self._set_stat_for_file("converted", filename, stats)
+
+    def get_stat_converted_for_file(self, filename):
+        """ get stat info for filename (after jpeg conversion)
+            returns: tuple of (mode, size, mtime) 
+        """
+        return self._get_stat_for_file("converted", filename)
+
     def get_info_for_uuid(self, uuid):
         """ returns the info JSON struct for a UUID """
         conn = self._conn
@@ -262,7 +332,6 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             info = None
 
-        logging.debug(f"get_info: {uuid}, {info}")
         return info
 
     def set_info_for_uuid(self, uuid, info):
@@ -278,8 +347,6 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
-        logging.debug(f"set_info: {uuid}, {info}")
-
     def get_exifdata_for_file(self, filename):
         """ returns the exifdata JSON struct for a file """
         filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
@@ -296,7 +363,6 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
             exifdata = None
 
-        logging.debug(f"get_exifdata: {filename}, {exifdata}")
         return exifdata
 
     def set_exifdata_for_file(self, filename, exifdata):
@@ -313,9 +379,17 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
-        logging.debug(f"set_exifdata: {filename}, {exifdata}")
+    def set_data(
-
+        self,
-    def set_data(self, filename, uuid, orig_stat, exif_stat, info_json, exif_json):
+        filename,
+        uuid,
+        orig_stat,
+        exif_stat,
+        converted_stat,
+        edited_stat,
+        info_json,
+        exif_json,
+    ):
         """ sets all the data for file and uuid at once 
         """
         filename = str(pathlib.Path(filename).relative_to(self._path))
@@ -339,6 +413,14 @@ class ExportDB(ExportDB_ABC):
                 + "WHERE filepath_normalized = ?;",
                 (*exif_stat, filename_normalized),
             )
+            c.execute(
+                "INSERT OR REPLACE INTO converted(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                (filename_normalized, *converted_stat),
+            )
+            c.execute(
+                "INSERT OR REPLACE INTO edited(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                (filename_normalized, *edited_stat),
+            )
             c.execute(
                 "INSERT OR REPLACE INTO info(uuid, json_info) VALUES (?, ?);",
                 (uuid, info_json),
@@ -358,6 +440,37 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def _set_stat_for_file(self, table, filename, stats):
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        if len(stats) != 3:
+            raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
+
+        conn = self._conn
+        c = conn.cursor()
+        c.execute(
+            f"INSERT OR REPLACE INTO {table}(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+            (filename, *stats),
+        )
+        conn.commit()
+
+    def _get_stat_for_file(self, table, filename):
+        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        conn = self._conn
+        c = conn.cursor()
+        c.execute(
+            f"SELECT mode, size, mtime FROM {table} WHERE filepath_normalized = ?",
+            (filename,),
+        )
+        results = c.fetchone()
+        if results:
+            stats = results[0:3]
+            mtime = int(stats[2]) if stats[2] is not None else None
+            stats = (stats[0], stats[1], mtime)
+        else:
+            stats = (None, None, None)
+
+        return stats
+
     def _open_export_db(self, dbfile):
         """ open export database and return a db connection
             if dbfile does not exist, will create and initialize the database 
@@ -365,15 +478,24 @@ class ExportDB(ExportDB_ABC):
         """
 
         if not os.path.isfile(dbfile):
-            logging.debug(f"dbfile {dbfile} doesn't exist, creating it")
             conn = self._get_db_connection(dbfile)
             if conn:
                 self._create_db_tables(conn)
+                self.was_created = True
+                self.was_upgraded = ()
+                self.version = OSXPHOTOS_EXPORTDB_VERSION
             else:
                 raise Exception("Error getting connection to database {dbfile}")
         else:
-            logging.debug(f"dbfile {dbfile} exists, opening it")
             conn = self._get_db_connection(dbfile)
+            self.was_created = False
+            version_info = self._get_database_version(conn)
+            if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION:
+                self._create_db_tables(conn)
+                self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
+            else:
+                self.was_upgraded = ()
+            self.version = OSXPHOTOS_EXPORTDB_VERSION
 
         return conn
 
@@ -387,6 +509,13 @@ class ExportDB(ExportDB_ABC):
 
         return conn
 
+    def _get_database_version(self, conn):
+        """ return tuple of (osxphotos, exportdb) versions for database connection conn """
+        version_info = conn.execute(
+            "SELECT osxphotos, exportdb, max(id) FROM version"
+        ).fetchone()
+        return (version_info[0], version_info[1])
+
     def _create_db_tables(self, conn):
         """ create (if not already created) the necessary db tables for the export database
             conn: sqlite3 db connection 
@@ -427,9 +556,25 @@ class ExportDB(ExportDB_ABC):
                              filepath_normalized TEXT NOT NULL,
                              json_exifdata JSON 
                              ); """,
-            "sql_files_idx": """ CREATE UNIQUE INDEX idx_files_filepath_normalized on files (filepath_normalized); """,
+            "sql_edited_table": """ CREATE TABLE IF NOT EXISTS edited (
-            "sql_info_idx": """ CREATE UNIQUE INDEX idx_info_uuid on info (uuid); """,
+                              id INTEGER PRIMARY KEY,
-            "sql_exifdata_idx": """ CREATE UNIQUE INDEX idx_exifdata_filename on exifdata (filepath_normalized); """,
+                              filepath_normalized TEXT NOT NULL,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
+            "sql_converted_table": """ CREATE TABLE IF NOT EXISTS converted (
+                              id INTEGER PRIMARY KEY,
+                              filepath_normalized TEXT NOT NULL,
+                              mode INTEGER,
+                              size INTEGER,
+                              mtime REAL
+                              ); """,
+            "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
+            "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
+            "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
+            "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
+            "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
         }
         try:
             c = conn.cursor()
@@ -445,11 +590,10 @@ class ExportDB(ExportDB_ABC):
 
     def __del__(self):
         """ ensure the database connection is closed """
-        if self._conn:
+        try:
-            try:
+            self._conn.close()
-                self._conn.close()
+        except:
-            except Error as e:
+            pass
-                logging.warning(e)
 
     def _insert_run_info(self):
         dt = datetime.datetime.utcnow().isoformat()
@@ -488,18 +632,18 @@ class ExportDBInMemory(ExportDB):
 
     def _open_export_db(self, dbfile):
         """ open export database and return a db connection
-            if dbfile does not exist, will create and initialize the database 
             returns: connection to the database 
         """
         if not os.path.isfile(dbfile):
-            logging.debug(f"dbfile {dbfile} doesn't exist, creating in memory version")
             conn = self._get_db_connection()
             if conn:
                 self._create_db_tables(conn)
+                self.was_created = True
+                self.was_upgraded = ()
+                self.version = OSXPHOTOS_EXPORTDB_VERSION
             else:
                 raise Exception("Error getting connection to in-memory database")
         else:
-            logging.debug(f"dbfile {dbfile} exists, opening it and copying to memory")
             try:
                 conn = sqlite3.connect(dbfile)
             except Error as e:
@@ -516,6 +660,14 @@ class ExportDBInMemory(ExportDB):
             conn = sqlite3.connect(":memory:")
             conn.cursor().executescript(tempfile.read())
             conn.commit()
+            self.was_created = False
+            _, exportdb_ver = self._get_database_version(conn)
+            if exportdb_ver < OSXPHOTOS_EXPORTDB_VERSION:
+                self._create_db_tables(conn)
+                self.was_upgraded = (exportdb_ver, OSXPHOTOS_EXPORTDB_VERSION)
+            else:
+                self.was_upgraded = ()
+            self.version = OSXPHOTOS_EXPORTDB_VERSION
 
         return conn
 

osxphotos/fileutil.py

@@ -8,6 +8,7 @@ import subprocess
 import sys
 from abc import ABC, abstractmethod
 
+from .imageconverter import ImageConverter
 
 class FileUtilABC(ABC):
     """ Abstract base class for FileUtil """
@@ -47,6 +48,11 @@ class FileUtilABC(ABC):
     def file_sig(cls, file1):
         pass
 
+    @classmethod
+    @abstractmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        pass
+
 
 class FileUtilMacOS(FileUtilABC):
     """ Various file utilities """
@@ -164,6 +170,21 @@ class FileUtilMacOS(FileUtilABC):
         """ return os.stat signature for file f1 """
         return cls._sig(os.stat(f1))
     
+    @classmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        """ converts image file src_file to jpeg format as dest_file
+
+            Args:
+                src_file: image file to convert
+                dest_file: destination path to write converted file to
+                compression quality: JPEG compression quality in range 0.0 <= compression_quality <= 1.0; default 1.0 (best quality)
+                
+            Returns:
+                True if success, otherwise False 
+        """
+        converter = ImageConverter()
+        return converter.write_jpeg(src_file, dest_file, compression_quality=compression_quality)
+
     @staticmethod
     def _sig(st):
         """ return tuple of (mode, size, mtime) of file based on os.stat
@@ -173,7 +194,6 @@ class FileUtilMacOS(FileUtilABC):
         # use int(st.st_mtime) because ditto does not copy fractional portion of mtime
         return (stat.S_IFMT(st.st_mode), st.st_size, int(st.st_mtime))
 
-
 class FileUtil(FileUtilMacOS):
     """ Various file utilities """
 
@@ -221,3 +241,7 @@ class FileUtilNoOp(FileUtil):
     def file_sig(cls, file1):
         cls.verbose(f"file_sig: {file1}")
         return (42, 42, 42)
+
+    @classmethod
+    def convert_to_jpeg(cls, src_file, dest_file, compression_quality=1.0):
+        cls.verbose(f"convert_to_jpeg: {src_file}, {dest_file}, {compression_quality}")

osxphotos/imageconverter.py

@@ -0,0 +1,112 @@
+""" ImageConverter class
+    Convert an image to JPEG using CoreImage -- 
+    for example, RAW to JPEG.  Only works if Mac equipped with GPU. """
+
+# reference: https://stackoverflow.com/questions/59330149/coreimage-ciimage-write-jpg-is-shifting-colors-macos/59334308#59334308
+
+import logging
+import pathlib
+
+import Metal
+import Quartz
+from Cocoa import NSURL
+from Foundation import NSDictionary
+
+# needed to capture system-level stderr
+from wurlitzer import pipes
+
+
+class ImageConverter:
+    """ Convert images to jpeg.  This class is a singleton
+        which will re-use the Core Image CIContext to avoid
+        creating a new context for every conversion. """
+
+    def __new__(cls, *args, **kwargs):
+        """ create new object or return instance of already created singleton """
+        if not hasattr(cls, "instance") or not cls.instance:
+            cls.instance = super().__new__(cls)
+
+        return cls.instance
+
+    def __init__(self):
+        """ return existing singleton or create a new one """
+
+        if hasattr(self, "context"):
+            return
+
+        """ initialize CIContext """
+        context_options = NSDictionary.dictionaryWithDictionary_(
+            {
+                "workingColorSpace": Quartz.CoreGraphics.kCGColorSpaceExtendedSRGB,
+                "workingFormat": Quartz.kCIFormatRGBAh,
+            }
+        )
+        mtldevice = Metal.MTLCreateSystemDefaultDevice()
+        self.context = Quartz.CIContext.contextWithMTLDevice_options_(
+            mtldevice, context_options
+        )
+
+    def write_jpeg(self, input_path, output_path, compression_quality=1.0):
+        """ convert image to jpeg and write image to output_path
+
+        Args:
+            input_path: path to input image (e.g. '/path/to/import/file.CR2') as str or pathlib.Path
+            output_path: path to exported jpeg (e.g. '/path/to/export/file.jpeg') as str or pathlib.Path
+            compression_quality: JPEG compression quality, float in range 0.0 to 1.0; default is 1.0 (best quality)
+
+        Return:
+            True if conversion successful, else False
+
+        Raises:
+            ValueError if compression quality not in range 0.0 to 1.0
+            FileNotFoundError if input_path doesn't exist
+        """
+
+        # accept input_path or output_path as pathlib.Path
+        if not isinstance(input_path, str):
+            input_path = str(input_path)
+
+        if not isinstance(output_path, str):
+            output_path = str(output_path)
+
+        if not pathlib.Path(input_path).is_file():
+            raise FileNotFoundError(f"could not find {input_path}")
+
+        if not (0.0 <= compression_quality <= 1.0):
+            raise ValueError(
+                "illegal value for compression_quality: {compression_quality}"
+            )
+
+        input_url = NSURL.fileURLWithPath_(input_path)
+        output_url = NSURL.fileURLWithPath_(output_path)
+
+        with pipes() as (out, err):
+            # capture stdout and stderr from system calls
+            # otherwise, Quartz.CIImage.imageWithContentsOfURL_
+            # prints to stderr something like:
+            # 2020-09-20 20:55:25.538 python[73042:5650492] Creating client/daemon connection: B8FE995E-3F27-47F4-9FA8-559C615FD774
+            # 2020-09-20 20:55:25.652 python[73042:5650492] Got the query meta data reply for: com.apple.MobileAsset.RawCamera.Camera, response: 0
+            input_image = Quartz.CIImage.imageWithContentsOfURL_(input_url)
+
+        if input_image is None:
+            logging.debug(f"Could not create CIImage for {input_path}")
+            return False
+
+        output_colorspace = input_image.colorSpace() or Quartz.CGColorSpaceCreateWithName(
+            Quartz.CoreGraphics.kCGColorSpaceSRGB
+        )
+
+        output_options = NSDictionary.dictionaryWithDictionary_(
+            {"kCGImageDestinationLossyCompressionQuality": compression_quality}
+        )
+        _, error = self.context.writeJPEGRepresentationOfImage_toURL_colorSpace_options_error_(
+            input_image, output_url, output_colorspace, output_options, None
+        )
+        if not error:
+            return True
+        else:
+            logging.debug(
+                "Error converting file {input_path} to jpeg at {output_path}: {error}"
+            )
+            return False
+

osxphotos/path_utils.py

@@ -0,0 +1,78 @@
+""" utility functions for validating/sanitizing path components """
+
+from ._constants import MAX_DIRNAME_LEN, MAX_FILENAME_LEN
+import pathvalidate
+
+
+def sanitize_filepath(filepath):
+    """ sanitize a filepath """
+    return pathvalidate.sanitize_filepath(filepath, platform="macos")
+
+
+def is_valid_filepath(filepath):
+    """ returns True if a filepath is valid otherwise False """
+    return pathvalidate.is_valid_filepath(filepath, platform="macos")
+
+
+def sanitize_filename(filename, replacement=":"):
+    """ replace any illegal characters in a filename and truncate filename if needed
+
+    Args:
+        filename: str, filename to sanitze
+        replacement: str, value to replace any illegal characters with; default = ":"
+    
+    Returns:
+        filename with any illegal characters replaced by replacement and truncated if necessary
+    """
+
+    if filename:
+        filename = filename.replace("/", replacement)
+        if len(filename) > MAX_FILENAME_LEN:
+            parts = filename.split(".")
+            drop = len(filename) - MAX_FILENAME_LEN
+            if len(parts) > 1:
+                # has an extension
+                ext = parts.pop(-1)
+                stem = ".".join(parts)
+                if drop > len(stem):
+                    ext = ext[:-drop]
+                else:
+                    stem = stem[:-drop]
+                filename = f"{stem}.{ext}"
+            else:
+                filename = filename[:-drop]
+    return filename
+
+
+def sanitize_dirname(dirname, replacement=":"):
+    """ replace any illegal characters in a directory name and truncate directory name if needed
+
+    Args:
+        dirname: str, directory name to sanitze
+        replacement: str, value to replace any illegal characters with; default = ":"
+    
+    Returns:
+        dirname with any illegal characters replaced by replacement and truncated if necessary
+    """
+    if dirname:
+        dirname = sanitize_pathpart(dirname, replacement=replacement)
+    return dirname
+
+
+def sanitize_pathpart(pathpart, replacement=":"):
+    """ replace any illegal characters in a path part (either directory or filename without extension) and truncate name if needed
+
+    Args:
+        pathpart: str, path part to sanitze
+        replacement: str, value to replace any illegal characters with; default = ":"
+    
+    Returns:
+        pathpart with any illegal characters replaced by replacement and truncated if necessary
+    """
+    if pathpart:
+        pathpart = pathpart.replace("/", replacement)
+        if len(pathpart) > MAX_DIRNAME_LEN:
+            drop = len(pathpart) - MAX_DIRNAME_LEN
+            pathpart = pathpart[:-drop]
+    return pathpart
+

osxphotos/personinfo.py

@@ -66,10 +66,10 @@ class PersonInfo:
             # no faces
             return []
 
-    def json(self):
+    def asdict(self):
-        """ Returns JSON representation of class instance """
+        """ Returns dictionary representation of class instance """
         keyphoto = self.keyphoto.uuid if self.keyphoto is not None else None
-        person = {
+        return {
             "uuid": self.uuid,
             "name": self.name,
             "displayname": self.display_name,
@@ -77,7 +77,10 @@ class PersonInfo:
             "facecount": self.facecount,
             "keyphoto": keyphoto,
         }
-        return json.dumps(person)
+
+    def json(self):
+        """ Returns JSON representation of class instance """
+        return json.dumps(self.asdict())
 
     def __str__(self):
         return f"PersonInfo(name={self.name}, display_name={self.display_name}, uuid={self.uuid}, facecount={self.facecount})"

osxphotos/photoinfo/_photoinfo_comments.py

@@ -0,0 +1,17 @@
+""" PhotoInfo methods to expose comments and likes for shared photos """ 
+
+@property
+def comments(self):
+    """ Returns list of Comment objects for any comments on the photo (sorted by date) """
+    try:
+        return self._db._db_comments_uuid[self.uuid]["comments"]
+    except:
+        return []
+
+@property
+def likes(self):
+    """ Returns list of Like objects for any likes on the photo (sorted by date) """
+    try:
+        return self._db._db_comments_uuid[self.uuid]["likes"]
+    except:
+        return []

osxphotos/photoinfo/_photoinfo_export.py

@@ -30,7 +30,7 @@ from .._constants import (
     _UNKNOWN_PERSON,
     _XMP_TEMPLATE_NAME,
 )
-from .._export_db import ExportDBNoOp
+from ..export_db import ExportDBNoOp
 from ..exiftool import ExifTool
 from ..fileutil import FileUtil
 from ..utils import dd_to_dms_str, findfiles
@@ -306,6 +306,8 @@ def export2(
     fileutil=FileUtil,
     dry_run=False,
     touch_file=False,
+    convert_to_jpeg=False,
+    jpeg_quality=1.0,
 ):
     """ export photo, like export but with update and dry_run options
         dest: must be valid destination path or exception raised 
@@ -313,10 +315,8 @@ def export2(
                     **NOTE**: if provided, user must ensure file extension (suffix) is correct. 
                     For example, if photo is .CR2 file, edited image may be .jpeg.  
                     If you provide an extension different than what the actual file is, 
-                    export will print a warning but will export the photo using the 
+                    will export the photo using the incorrect file extension (unless use_photos_export is true, 
-                    incorrect file extension (unless use_photos_export is true, in which case export will
+                    in which case export will use the extension provided by Photos upon export.
-                    use the extension provided by Photos upon export; in this case, an incorrect extension is
-                    silently ignored).
                     e.g. to get the extension of the edited photo, 
                     reference PhotoInfo.path_edited
         edited: (boolean, default=False); if True will export the edited version of the photo 
@@ -335,7 +335,6 @@ def export2(
         timeout: (int, default=120) timeout in seconds used with use_photos_export
         exiftool: (boolean, default = False); if True, will use exiftool to write metadata to export file
         no_xattr: (boolean, default = False); if True, exports file without preserving extended attributes
-        returns list of full paths to the exported files
         use_albums_as_keywords: (boolean, default = False); if True, will include album names in keywords
         when exporting metadata with exiftool or sidecar
         use_persons_as_keywords: (boolean, default = False); if True, will include person names in keywords
@@ -349,6 +348,8 @@ def export2(
         fileutil: (FileUtilABC); class that conforms to FileUtilABC with various file utilities
         dry_run: (boolean, default=False); set to True to run in "dry run" mode
         touch_file: (boolean, default=False); if True, sets file's modification time upon photo date
+        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
         Returns: ExportResults namedtuple with fields: exported, new, updated, skipped 
                     where each field is a list of file paths
@@ -357,6 +358,10 @@ def export2(
               and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp)
             """
 
+    # NOTE: This function is very complex and does a lot of things.
+    # Don't modify this code if you don't fully understand everything it does.
+    # TODO: This is a good candidate for refactoring.
+
     # when called from export(), won't get an export_db, so use no-op version
     if export_db is None:
         export_db = ExportDBNoOp()
@@ -392,34 +397,41 @@ def export2(
         raise TypeError(
             "Too many positional arguments.  Should be at most two: destination, filename."
         )
-    else:
-        # verify destination is a valid path
-        if dest is None:
-            raise ValueError("Destination must not be None")
-        elif not dry_run and not os.path.isdir(dest):
-            raise FileNotFoundError("Invalid path passed to export")
 
-        if filename and len(filename) == 1:
+    # verify destination is a valid path
-            # if filename passed, use it
+    if dest is None:
-            fname = filename[0]
+        raise ValueError("Destination must not be None")
-        else:
+    elif not dry_run and not os.path.isdir(dest):
-            # no filename provided so use the default
+        raise FileNotFoundError("Invalid path passed to export")
-            # if edited file requested, use filename but add _edited
+
-            # need to use file extension from edited file as Photos saves a jpeg once edited
+    if filename and len(filename) == 1:
-            if edited and not use_photos_export:
+        # if filename passed, use it
-                # verify we have a valid path_edited and use that to get filename
+        fname = filename[0]
-                if not self.path_edited:
+    else:
-                    raise FileNotFoundError(
+        # no filename provided so use the default
-                        "edited=True but path_edited is none; hasadjustments: "
+        # if edited file requested, use filename but add _edited
-                        f" {self.hasadjustments}"
+        # need to use file extension from edited file as Photos saves a jpeg once edited
-                    )
+        if edited and not use_photos_export:
-                edited_name = pathlib.Path(self.path_edited).name
+            # verify we have a valid path_edited and use that to get filename
-                edited_suffix = pathlib.Path(edited_name).suffix
+            if not self.path_edited:
-                fname = (
+                raise FileNotFoundError(
-                    pathlib.Path(self.filename).stem + edited_identifier + edited_suffix
+                    "edited=True but path_edited is none; hasadjustments: "
+                    f" {self.hasadjustments}"
                 )
-            else:
+            edited_name = pathlib.Path(self.path_edited).name
-                fname = self.filename
+            edited_suffix = pathlib.Path(edited_name).suffix
+            fname = pathlib.Path(self.filename).stem + edited_identifier + edited_suffix
+        else:
+            fname = self.filename
+
+    uti = self.uti if edited else self.uti_original
+    if convert_to_jpeg and self.isphoto and uti != "public.jpeg":
+        # not a jpeg but will convert to jpeg upon export so fix file extension
+        fname_new = pathlib.Path(fname)
+        fname = str(fname_new.parent / f"{fname_new.stem}.jpeg")
+    else:
+        # nothing to convert
+        convert_to_jpeg = False
 
     # check destination path
     dest = pathlib.Path(dest)
@@ -473,16 +485,12 @@ def export2(
             raise FileNotFoundError(f"{src} does not appear to exist")
 
         if not _check_export_suffix(src, dest, edited):
-            logging.warning(
+            logging.debug(
                 f"Invalid destination suffix: {dest.suffix} for {self.path}, "
                 + f"edited={edited}, path_edited={self.path_edited}, "
                 + f"original_filename={self.original_filename}, filename={self.filename}"
             )
 
-        logging.debug(
-            f"exporting {src} to {dest}, overwrite={overwrite}, increment={increment}, dest exists: {dest.exists()}"
-        )
-
         # found source now try to find right destination
         if update and dest.exists():
             # destination exists, check to see if destination is the right UUID
@@ -498,14 +506,13 @@ def export2(
                     self.uuid,
                     fileutil.file_sig(dest),
                     (None, None, None),
+                    (None, None, None),
+                    (None, None, None),
                     self.json(),
                     None,
                 )
             if dest_uuid != self.uuid:
                 # not the right file, find the right one
-                logging.debug(
-                    f"Need to find right photo: uuid={self.uuid}, dest={dest_uuid}, dest={dest}, path={self.path}"
-                )
                 count = 1
                 glob_str = str(dest.parent / f"{dest.stem} (*{dest.suffix}")
                 dest_files = glob.glob(glob_str)
@@ -513,17 +520,11 @@ def export2(
                 for file_ in dest_files:
                     dest_uuid = export_db.get_uuid_for_file(file_)
                     if dest_uuid == self.uuid:
-                        logging.debug(
-                            f"Found matching file for uuid: {dest_uuid}, {file_}"
-                        )
                         dest = pathlib.Path(file_)
                         found_match = True
                         break
                     elif dest_uuid is None and fileutil.cmp(src, file_):
                         # files match, update the UUID
-                        logging.debug(
-                            f"Found matching file with blank uuid: {self.uuid}, {file_}"
-                        )
                         dest = pathlib.Path(file_)
                         found_match = True
                         export_db.set_data(
@@ -531,16 +532,14 @@ def export2(
                             self.uuid,
                             fileutil.file_sig(dest),
                             (None, None, None),
+                            (None, None, None),
+                            (None, None, None),
                             self.json(),
                             None,
                         )
                         break
 
                 if not found_match:
-                    logging.debug(
-                        f"Didn't find destination match for uuid {self.uuid} {dest}"
-                    )
-
                     # increment the destination file
                     count = 1
                     glob_str = str(dest.parent / f"{dest.stem}*")
@@ -551,7 +550,6 @@ def export2(
                         dest_new = f"{dest.stem} ({count})"
                         count += 1
                     dest = dest.parent / f"{dest_new}{dest.suffix}"
-                    logging.debug(f"New destination = {dest}, uuid = {self.uuid}")
 
         # export the dest file
         results = self._export_photo(
@@ -564,7 +562,10 @@ def export2(
             export_as_hardlink,
             exiftool,
             touch_file,
-            fileutil,
+            convert_to_jpeg,
+            fileutil=fileutil,
+            edited=edited,
+            jpeg_quality=jpeg_quality,
         )
         exported_files = results.exported
         update_new_files = results.new
@@ -591,7 +592,8 @@ def export2(
                     export_as_hardlink,
                     exiftool,
                     touch_file,
-                    fileutil,
+                    False,
+                    fileutil=fileutil,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
@@ -618,7 +620,9 @@ def export2(
                     export_as_hardlink,
                     exiftool,
                     touch_file,
-                    fileutil,
+                    convert_to_jpeg,
+                    fileutil=fileutil,
+                    jpeg_quality=jpeg_quality,
                 )
                 exported_files.extend(results.exported)
                 update_new_files.extend(results.new)
@@ -632,8 +636,11 @@ def export2(
         exported = []
         # export live_photo .mov file?
         live_photo = True if live_photo and self.live_photo else False
-        if edited:
+        if edited or self.shared:
             # exported edited version and not original
+            # shared photos (in shared albums) show up as not having adjustments (not edited)
+            # but Photos is unable to export the "original" as only a jpeg copy is shared in iCloud
+            # so tell Photos to export the current version in this case
             if filename:
                 # use filename stem provided
                 filestem = dest.stem
@@ -667,7 +674,6 @@ def export2(
                 burst=self.burst,
                 dry_run=dry_run,
             )
-
         if exported:
             if touch_file:
                 for exported_file in exported:
@@ -683,6 +689,7 @@ def export2(
                 f"Error exporting photo {self.uuid} to {dest} with use_photos_export"
             )
 
+    # export metadata
     if sidecar_json:
         logging.debug("writing exiftool_json_sidecar")
         sidecar_filename = dest.parent / pathlib.Path(f"{dest.stem}{dest.suffix}.json")
@@ -744,7 +751,6 @@ def export2(
             if old_data is None or files_are_different:
                 # didn't have old data, assume we need to write it
                 # or files were different
-                logging.debug(f"No exifdata for {exported_file}, writing it")
                 if not dry_run:
                     self._write_exif_data(
                         exported_file,
@@ -768,7 +774,6 @@ def export2(
                 exif_files_updated.append(exported_file)
     elif exiftool and exif_files:
         for exported_file in exif_files:
-            logging.debug(f"Writing exif data to {exported_file}")
             if not dry_run:
                 self._write_exif_data(
                     exported_file,
@@ -822,7 +827,10 @@ def _export_photo(
     export_as_hardlink,
     exiftool,
     touch_file,
+    convert_to_jpeg,
     fileutil=FileUtil,
+    edited=False,
+    jpeg_quality=1.0,
 ):
     """ Helper function for export()
         Does the actual copy or hardlink taking the appropriate 
@@ -840,12 +848,21 @@ def _export_photo(
         export_as_hardlink: bool
         exiftool: bool
         touch_file: bool
+        convert_to_jpeg: bool; if True, convert file to jpeg on export
         fileutil: FileUtil class that conforms to fileutil.FileUtilABC
+        edited: bool; set to True if exporting edited version of photo
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
 
     Returns:
         ExportResults
+
+    Raises:
+        ValueError if export_as_hardlink and convert_to_jpeg both True
     """
 
+    if export_as_hardlink and convert_to_jpeg:
+        raise ValueError("export_as_hardlink and convert_to_jpeg cannot both be True")
+
     exported_files = []
     update_updated_files = []
     update_new_files = []
@@ -854,40 +871,44 @@ def _export_photo(
 
     dest_str = str(dest)
     dest_exists = dest.exists()
-    if export_as_hardlink:
+    op_desc = "export_as_hardlink" if export_as_hardlink else "export_by_copying"
-        op_desc = "export_as_hardlink"
-    else:
-        op_desc = "export_by_copying"
 
-    if not update:
+    if update:  # updating
-        # not update, export the file
+        cmp_touch, cmp_orig = False, False
-        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
+        if dest_exists:
-        exported_files.append(dest_str)
-        if touch_file:
-            sig = fileutil.file_sig(src)
-            sig = (sig[0], sig[1], int(self.date.timestamp()))
-            if not fileutil.cmp_file_sig(src, sig):
-                touched_files.append(dest_str)
-    else:  # updating
-        if not dest_exists:
-            # update, destination doesn't exist (new file)
-            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
-            update_new_files.append(dest_str)
-            if touch_file:
-                touched_files.append(dest_str)
-        else:
             # update, destination exists, but we might not need to replace it...
             if exiftool:
                 sig_exif = export_db.get_stat_exif_for_file(dest_str)
                 cmp_orig = fileutil.cmp_file_sig(dest_str, sig_exif)
                 sig_exif = (sig_exif[0], sig_exif[1], int(self.date.timestamp()))
                 cmp_touch = fileutil.cmp_file_sig(dest_str, sig_exif)
+            elif convert_to_jpeg:
+                sig_converted = export_db.get_stat_converted_for_file(dest_str)
+                cmp_orig = fileutil.cmp_file_sig(dest_str, sig_converted)
+                sig_converted = (
+                    sig_converted[0],
+                    sig_converted[1],
+                    int(self.date.timestamp()),
+                )
+                cmp_touch = fileutil.cmp_file_sig(dest_str, sig_converted)
             else:
                 cmp_orig = fileutil.cmp(src, dest)
                 cmp_touch = fileutil.cmp(src, dest, mtime1=int(self.date.timestamp()))
 
             sig_cmp = cmp_touch if touch_file else cmp_orig
 
+            if edited:
+                # requested edited version of photo
+                # need to see if edited version in Photos library has changed
+                # (e.g. it's been edited again)
+                sig_edited = export_db.get_stat_edited_for_file(dest_str)
+                cmp_edited = (
+                    fileutil.cmp_file_sig(src, sig_edited)
+                    if sig_edited != (None, None, None)
+                    else False
+                )
+                sig_cmp = sig_cmp and cmp_edited
+
             if (export_as_hardlink and dest.samefile(src)) or (
                 not export_as_hardlink and not dest.samefile(src) and sig_cmp
             ):
@@ -911,7 +932,24 @@ def _export_photo(
                     if touch_file:
                         touched_files.append(dest_str)
 
+        else:
+            # update, destination doesn't exist (new file)
+            logging.debug(f"Update: exporting new file with {op_desc} {src} {dest}")
+            update_new_files.append(dest_str)
+            if touch_file:
+                touched_files.append(dest_str)
+    else:
+        # not update, export the file
+        logging.debug(f"Exporting file with {op_desc} {src} {dest}")
+        exported_files.append(dest_str)
+        if touch_file:
+            sig = fileutil.file_sig(src)
+            sig = (sig[0], sig[1], int(self.date.timestamp()))
+            if not fileutil.cmp_file_sig(src, sig):
+                touched_files.append(dest_str)
     if not update_skipped_files:
+        converted_stat = (None, None, None)
+        edited_stat = fileutil.file_sig(src) if edited else (None, None, None)
         if dest_exists and (update or overwrite):
             # need to remove the destination first
             logging.debug(
@@ -920,6 +958,10 @@ def _export_photo(
             fileutil.unlink(dest)
         if export_as_hardlink:
             fileutil.hardlink(src, dest)
+        elif convert_to_jpeg:
+            # use convert_to_jpeg to export the file
+            fileutil.convert_to_jpeg(src, dest_str, compression_quality=jpeg_quality)
+            converted_stat = fileutil.file_sig(dest_str)
         else:
             fileutil.copy(src, dest_str, norsrc=no_xattr)
 
@@ -928,6 +970,8 @@ def _export_photo(
             self.uuid,
             fileutil.file_sig(dest_str),
             (None, None, None),
+            converted_stat,
+            edited_stat,
             self.json(),
             None,
         )
@@ -1085,11 +1129,10 @@ def _exiftool_json_sidecar(
 
     (lat, lon) = self.location
     if lat is not None and lon is not None:
-        lat_str, lon_str = dd_to_dms_str(lat, lon)
+        exif["EXIF:GPSLatitude"] = lat
-        exif["EXIF:GPSLatitude"] = lat_str
+        exif["EXIF:GPSLongitude"] = lon
-        exif["EXIF:GPSLongitude"] = lon_str
+        lat_ref = "N" if lat >= 0 else "S"
-        lat_ref = "North" if lat >= 0 else "South"
+        lon_ref = "E" if lon >= 0 else "W"
-        lon_ref = "East" if lon >= 0 else "West"
         exif["EXIF:GPSLatitudeRef"] = lat_ref
         exif["EXIF:GPSLongitudeRef"] = lon_ref
 

osxphotos/photoinfo/photoinfo.py

@@ -5,6 +5,7 @@ PhotosDB.photos() returns a list of PhotoInfo objects
 """
 
 import dataclasses
+import datetime
 import json
 import logging
 import os
@@ -20,6 +21,7 @@ from .._constants import (
     _PHOTOS_4_ALBUM_KIND,
     _PHOTOS_4_ROOT_FOLDER,
     _PHOTOS_4_VERSION,
+    _PHOTOS_5_VERSION,
     _PHOTOS_5_ALBUM_KIND,
     _PHOTOS_5_IMPORT_SESSION_ALBUM_KIND,
     _PHOTOS_5_SHARED_ALBUM_KIND,
@@ -58,6 +60,7 @@ class PhotoInfo:
         ExportResults,
     )
     from ._photoinfo_scoreinfo import score, ScoreInfo
+    from ._photoinfo_comments import comments, likes 
 
     def __init__(self, db=None, uuid=None, info=None):
         self._uuid = uuid
@@ -67,11 +70,9 @@ class PhotoInfo:
     @property
     def filename(self):
         """ filename of the picture """
-        # sourcery off
+        if self._db._db_version <= _PHOTOS_4_VERSION and self.has_raw and self.raw_original:
-        if self.has_raw and self.raw_original:
+            # return the JPEG version as that's what Photos 5+ does
-            # return name of the RAW file
+            return self._info["raw_pair_info"]["filename"]
-            # TODO: not yet implemented
-            return self._info["filename"]
         else:
             return self._info["filename"]
 
@@ -79,7 +80,11 @@ class PhotoInfo:
     def original_filename(self):
         """ original filename of the picture 
             Photos 5 mangles filenames upon import """
-        return self._info["originalFilename"]
+        if self._db._db_version <= _PHOTOS_4_VERSION and self.has_raw and self.raw_original:
+            # return the JPEG version as that's what Photos 5+ does
+            return self._info["raw_pair_info"]["originalFilename"]
+        else:
+            return self._info["originalFilename"]
 
     @property
     def date(self):
@@ -107,39 +112,67 @@ class PhotoInfo:
     @property
     def path(self):
         """ absolute path on disk of the original picture """
+        try:
+            return self._path
+        except AttributeError:
+            self._path = None
+            photopath = None
+            if self._info["isMissing"] == 1:
+                return photopath  # path would be meaningless until downloaded
 
-        photopath = None
+            if self._db._db_version <= _PHOTOS_4_VERSION:
-        if self._info["isMissing"] == 1:
+                if self._info["has_raw"]:
-            return photopath  # path would be meaningless until downloaded
+                    # return the path to JPEG even if RAW is original
+                    vol = (
+                        self._db._dbvolumes[self._info["raw_pair_info"]["volumeId"]]
+                        if self._info["raw_pair_info"]["volumeId"] is not None
+                        else None
+                    )
+                    if vol is not None:
+                        photopath = os.path.join(
+                            "/Volumes", vol, self._info["raw_pair_info"]["imagePath"]
+                        )
+                    else:
+                        photopath = os.path.join(
+                            self._db._masters_path,
+                            self._info["raw_pair_info"]["imagePath"],
+                        )
+                else:
+                    vol = self._info["volume"]
+                    if vol is not None:
+                        photopath = os.path.join(
+                            "/Volumes", vol, self._info["imagePath"]
+                        )
+                    else:
+                        photopath = os.path.join(
+                            self._db._masters_path, self._info["imagePath"]
+                        )
+                self._path = photopath
+                return photopath
 
-        if self._db._db_version <= _PHOTOS_4_VERSION:
+            if self._info["shared"]:
-            vol = self._info["volume"]
+                # shared photo
-            if vol is not None:
+                photopath = os.path.join(
-                photopath = os.path.join("/Volumes", vol, self._info["imagePath"])
+                    self._db._library_path,
+                    _PHOTOS_5_SHARED_PHOTO_PATH,
+                    self._info["directory"],
+                    self._info["filename"],
+                )
+                self._path = photopath
+                return photopath
+
+            if self._info["directory"].startswith("/"):
+                photopath = os.path.join(
+                    self._info["directory"], self._info["filename"]
+                )
             else:
                 photopath = os.path.join(
-                    self._db._masters_path, self._info["imagePath"]
+                    self._db._masters_path,
+                    self._info["directory"],
+                    self._info["filename"],
                 )
+            self._path = photopath
             return photopath
-            # TODO: Is there a way to use applescript or PhotoKit to force the download in this
-
-        if self._info["shared"]:
-            # shared photo
-            photopath = os.path.join(
-                self._db._library_path,
-                _PHOTOS_5_SHARED_PHOTO_PATH,
-                self._info["directory"],
-                self._info["filename"],
-            )
-            return photopath
-
-        if self._info["directory"].startswith("/"):
-            photopath = os.path.join(self._info["directory"], self._info["filename"])
-        else:
-            photopath = os.path.join(
-                self._db._masters_path, self._info["directory"], self._info["filename"]
-            )
-        return photopath
 
     @property
     def path_edited(self):
@@ -149,109 +182,132 @@ class PhotoInfo:
         # TODO: break this code into a _path_edited_4 and _path_edited_5
         # version to simplify the big if/then; same for path_live_photo
 
-        photopath = None
+        try:
-
+            return self._path_edited
-        if self._db._db_version <= _PHOTOS_4_VERSION:
+        except AttributeError:
-            if self._info["hasAdjustments"]:
+            if self._db._db_version <= _PHOTOS_4_VERSION:
-                edit_id = self._info["edit_resource_id"]
+                self._path_edited = self._path_edited_4()
-                if edit_id is not None:
+                return self._path_edited
-                    library = self._db._library_path
-                    folder_id, file_id = _get_resource_loc(edit_id)
-                    # todo: is this always true or do we need to search file file_id under folder_id
-                    # figure out what kind it is and build filename
-                    filename = None
-                    if self._info["type"] == _PHOTO_TYPE:
-                        # it's a photo
-                        filename = f"fullsizeoutput_{file_id}.jpeg"
-                    elif self._info["type"] == _MOVIE_TYPE:
-                        # it's a movie
-                        filename = f"fullsizeoutput_{file_id}.mov"
-                    else:
-                        # don't know what it is!
-                        logging.debug(f"WARNING: unknown type {self._info['type']}")
-                        return None
-
-                    # photopath appears to usually be in "00" subfolder but
-                    # could be elsewhere--I haven't figured out this logic yet
-                    # first see if it's in 00
-                    photopath = os.path.join(
-                        library,
-                        "resources",
-                        "media",
-                        "version",
-                        folder_id,
-                        "00",
-                        filename,
-                    )
-
-                    if not os.path.isfile(photopath):
-                        rootdir = os.path.join(
-                            library, "resources", "media", "version", folder_id
-                        )
-
-                        for dirname, _, filelist in os.walk(rootdir):
-                            if filename in filelist:
-                                photopath = os.path.join(dirname, filename)
-                                break
-
-                    # check again to see if we found a valid file
-                    if not os.path.isfile(photopath):
-                        logging.debug(
-                            f"MISSING PATH: edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
-                        )
-                        photopath = None
-                else:
-                    logging.debug(
-                        f"{self.uuid} hasAdjustments but edit_resource_id is None"
-                    )
-                    photopath = None
             else:
+                self._path_edited = self._path_edited_5()
+                return self._path_edited
+
+    def _path_edited_5(self):
+        """ return path_edited for Photos >= 5 """
+        # In Photos 5.0 / Catalina / MacOS 10.15:
+        # edited photos appear to always be converted to .jpeg and stored in
+        # library_name/resources/renders/X/UUID_1_201_a.jpeg
+        # where X = first letter of UUID
+        # and UUID = UUID of image
+        # this seems to be true even for photos not copied to Photos library and
+        # where original format was not jpg/jpeg
+        # if more than one edit, previous edit is stored as UUID_p.jpeg
+        #
+        # In Photos 6.0 / Big Sur, the edited image is a .heic if the photo isn't a jpeg,
+        # otherwise it's a jpeg.  It could also be a jpeg if photo library upgraded from earlier
+        # version.
+
+        if self._db._db_version < _PHOTOS_5_VERSION:
+            raise RuntimeError("Wrong database format!")
+
+        if self._info["hasAdjustments"]:
+            library = self._db._library_path
+            directory = self._uuid[0]  # first char of uuid
+            filename = None
+            if self._info["type"] == _PHOTO_TYPE:
+                # it's a photo
+                if self._db._photos_ver == 5:
+                    filename = f"{self._uuid}_1_201_a.jpeg"
+                else:
+                    # could be a heic or a jpeg
+                    if self.uti == "public.heic":
+                        filename = f"{self._uuid}_1_201_a.heic"
+                    else:
+                        filename = f"{self._uuid}_1_201_a.jpeg"
+            elif self._info["type"] == _MOVIE_TYPE:
+                # it's a movie
+                filename = f"{self._uuid}_2_0_a.mov"
+            else:
+                # don't know what it is!
+                logging.debug(f"WARNING: unknown type {self._info['type']}")
+                return None
+
+            photopath = os.path.join(
+                library, "resources", "renders", directory, filename
+            )
+
+            if not os.path.isfile(photopath):
+                logging.debug(
+                    f"edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
+                )
                 photopath = None
-
-            # if self._info["isMissing"] == 1:
-            #     photopath = None  # path would be meaningless until downloaded
         else:
-            # in Photos 5.0 / Catalina / MacOS 10.15:
+            photopath = None
-            # edited photos appear to always be converted to .jpeg and stored in
-            # library_name/resources/renders/X/UUID_1_201_a.jpeg
-            # where X = first letter of UUID
-            # and UUID = UUID of image
-            # this seems to be true even for photos not copied to Photos library and
-            # where original format was not jpg/jpeg
-            # if more than one edit, previous edit is stored as UUID_p.jpeg
 
-            if self._info["hasAdjustments"]:
+        # TODO: might be possible for original/master to be missing but edit to still be there
+        # if self._info["isMissing"] == 1:
+        #     photopath = None  # path would be meaningless until downloaded
+
+        # logging.debug(photopath)
+
+        return photopath
+
+    def _path_edited_4(self):
+        """ return path_edited for Photos <= 4 """
+
+        if self._db._db_version > _PHOTOS_4_VERSION:
+            raise RuntimeError("Wrong database format!")
+
+        photopath = None
+        if self._info["hasAdjustments"]:
+            edit_id = self._info["edit_resource_id"]
+            if edit_id is not None:
                 library = self._db._library_path
-                directory = self._uuid[0]  # first char of uuid
+                folder_id, file_id = _get_resource_loc(edit_id)
+                # todo: is this always true or do we need to search file file_id under folder_id
+                # figure out what kind it is and build filename
                 filename = None
                 if self._info["type"] == _PHOTO_TYPE:
                     # it's a photo
-                    filename = f"{self._uuid}_1_201_a.jpeg"
+                    filename = f"fullsizeoutput_{file_id}.jpeg"
                 elif self._info["type"] == _MOVIE_TYPE:
                     # it's a movie
-                    filename = f"{self._uuid}_2_0_a.mov"
+                    filename = f"fullsizeoutput_{file_id}.mov"
                 else:
                     # don't know what it is!
                     logging.debug(f"WARNING: unknown type {self._info['type']}")
                     return None
 
+                # photopath appears to usually be in "00" subfolder but
+                # could be elsewhere--I haven't figured out this logic yet
+                # first see if it's in 00
                 photopath = os.path.join(
-                    library, "resources", "renders", directory, filename
+                    library, "resources", "media", "version", folder_id, "00", filename,
                 )
 
+                if not os.path.isfile(photopath):
+                    rootdir = os.path.join(
+                        library, "resources", "media", "version", folder_id
+                    )
+
+                    for dirname, _, filelist in os.walk(rootdir):
+                        if filename in filelist:
+                            photopath = os.path.join(dirname, filename)
+                            break
+
+                # check again to see if we found a valid file
                 if not os.path.isfile(photopath):
                     logging.debug(
-                        f"edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
+                        f"MISSING PATH: edited file for UUID {self._uuid} should be at {photopath} but does not appear to exist"
                     )
                     photopath = None
             else:
+                logging.debug(
+                    f"{self.uuid} hasAdjustments but edit_resource_id is None"
+                )
                 photopath = None
-
+        else:
-            # TODO: might be possible for original/master to be missing but edit to still be there
+            photopath = None
-            # if self._info["isMissing"] == 1:
-            #     photopath = None  # path would be meaningless until downloaded
-
-            # logging.debug(photopath)
 
         return photopath
 
@@ -473,7 +529,40 @@ class PhotoInfo:
         """ Returns Uniform Type Identifier (UTI) for the image
             for example: public.jpeg or com.apple.quicktime-movie
         """
-        return self._info["UTI"]
+        if self._db._db_version <= _PHOTOS_4_VERSION:
+            if self.hasadjustments:
+                return self._info["UTI_edited"]
+            elif self.has_raw and self.raw_original:
+                # return UTI of the non-raw image to match Photos 5+ behavior
+                return self._info["raw_pair_info"]["UTI"]
+            else:
+                return self._info["UTI"]
+        else:
+            return self._info["UTI"]
+
+    @property
+    def uti_original(self):
+        """ Returns Uniform Type Identifier (UTI) for the original image
+            for example: public.jpeg or com.apple.quicktime-movie
+        """
+        if self._db._db_version <= _PHOTOS_4_VERSION and self._info["has_raw"]:
+            return self._info["raw_pair_info"]["UTI"]
+        elif self.shared:
+            # TODO: need reliable way to get original UTI for shared
+            return self.uti
+        else:
+            return self._info["UTI_original"]
+
+    @property
+    def uti_edited(self):
+        """ Returns Uniform Type Identifier (UTI) for the edited image 
+            if the photo has been edited, otherwise None; 
+            for example: public.jpeg 
+        """
+        if self._db._db_version >= _PHOTOS_5_VERSION:
+            return self.uti if self.hasadjustments else None
+        else:
+            return self._info["UTI_edited"]
 
     @property
     def uti_raw(self):
@@ -664,12 +753,17 @@ class PhotoInfo:
 
     @property
     def has_raw(self):
-        """ returns True if photo has an associated RAW image, otherwise False """
+        """ returns True if photo has an associated raw image (that is, it's a RAW+JPEG pair), otherwise False """
         return self._info["has_raw"]
 
+    @property
+    def israw(self):
+        """ returns True if photo is a raw image. For images with an associated RAW+JPEG pair, see has_raw """
+        return "raw-image" in self.uti_original
+
     @property
     def raw_original(self):
-        """ returns True if associated RAW image and the RAW image is selected in Photos
+        """ returns True if associated raw image and the raw image is selected in Photos
             via "Use RAW as Original "
             otherwise returns False """
         return self._info["raw_is_original"]
@@ -716,6 +810,9 @@ class PhotoInfo:
         path_sep=None,
         expand_inplace=False,
         inplace_sep=None,
+        filename=False,
+        dirname=False,
+        replacement=":",
     ):
         """Renders a template string for PhotoInfo instance using PhotoTemplate
 
@@ -728,6 +825,9 @@ class PhotoInfo:
                 instead of returning individual strings
             inplace_sep: optional string to use as separator between multi-valued keywords
                 with expand_inplace; default is ','
+            filename: if True, template output will be sanitized to produce valid file name
+            dirname: if True, template output will be sanitized to produce valid directory name 
+            replacement: str, value to replace any illegal file path characters with; default = ":"
         
         Returns:
             ([rendered_strings], [unmatched]): tuple of list of rendered strings and list of unmatched template values
@@ -739,6 +839,9 @@ class PhotoInfo:
             path_sep=path_sep,
             expand_inplace=expand_inplace,
             inplace_sep=inplace_sep,
+            filename=filename,
+            dirname=dirname,
+            replacement=replacement
         )
 
     @property
@@ -848,22 +951,23 @@ class PhotoInfo:
         }
         return yaml.dump(info, sort_keys=False)
 
-    def json(self):
+    def asdict(self):
-        """ return JSON representation """
+        """ return dict representation """
 
-        date_modified_iso = (
-            self.date_modified.isoformat() if self.date_modified else None
-        )
         folders = {album.title: album.folder_names for album in self.album_info}
         exif = dataclasses.asdict(self.exif_info) if self.exif_info else {}
-        place = self.place.as_dict() if self.place else {}
+        place = self.place.asdict() if self.place else {}
         score = dataclasses.asdict(self.score) if self.score else {}
+        comments = [comment.asdict() for comment in self.comments]
+        likes = [like.asdict() for like in self.likes]
+        faces = [face.asdict() for face in self.face_info]
 
-        pic = {
+        return {
+            "library": self._db._library_path,
             "uuid": self.uuid,
             "filename": self.filename,
             "original_filename": self.original_filename,
-            "date": self.date.isoformat(),
+            "date": self.date,
             "description": self.description,
             "title": self.title,
             "keywords": self.keywords,
@@ -872,6 +976,7 @@ class PhotoInfo:
             "albums": self.albums,
             "folders": folders,
             "persons": self.persons,
+            "faces": faces,
             "path": self.path,
             "ismissing": self.ismissing,
             "hasadjustments": self.hasadjustments,
@@ -885,12 +990,13 @@ class PhotoInfo:
             "isphoto": self.isphoto,
             "ismovie": self.ismovie,
             "uti": self.uti,
+            "uti_original": self.uti_original,
             "burst": self.burst,
             "live_photo": self.live_photo,
             "path_live_photo": self.path_live_photo,
             "iscloudasset": self.iscloudasset,
             "incloud": self.incloud,
-            "date_modified": date_modified_iso,
+            "date_modified": self.date_modified,
             "portrait": self.portrait,
             "screenshot": self.screenshot,
             "slow_mo": self.slow_mo,
@@ -899,6 +1005,8 @@ class PhotoInfo:
             "selfie": self.selfie,
             "panorama": self.panorama,
             "has_raw": self.has_raw,
+            "israw": self.israw,
+            "raw_original": self.raw_original,
             "uti_raw": self.uti_raw,
             "path_raw": self.path_raw,
             "place": place,
@@ -912,8 +1020,17 @@ class PhotoInfo:
             "original_width": self.original_width,
             "original_orientation": self.original_orientation,
             "original_filesize": self.original_filesize,
+            "comments": comments,
+            "likes": likes,
         }
-        return json.dumps(pic)
+
+    def json(self):
+        """ Return JSON representation """
+        def default(o):
+            if isinstance(o, (datetime.date, datetime.datetime)):
+                return o.isoformat()
+
+        return json.dumps(self.asdict(), sort_keys=True, default=default)
 
     def __eq__(self, other):
         """ Compare two PhotoInfo objects for equality """

osxphotos/photosdb/_photosdb_process_comments.py

@@ -0,0 +1,157 @@
+""" PhotosDB method for processing comments and likes on shared photos.
+    Do not import this module directly """
+
+import dataclasses
+import datetime
+from dataclasses import dataclass
+
+from .._constants import _DB_TABLE_NAMES, _PHOTOS_4_VERSION, TIME_DELTA
+from ..utils import _open_sql_file, normalize_unicode
+
+
+def _process_comments(self):
+    """ load the comments and likes data from the database 
+        this is a PhotosDB method that should be imported in 
+        the PhotosDB class definition in photosdb.py
+    """
+    self._db_hashed_person_id = {}
+    self._db_comments_uuid = {}
+    if self._db_version <= _PHOTOS_4_VERSION:
+        _process_comments_4(self)
+    else:
+        _process_comments_5(self)
+
+
+@dataclass
+class CommentInfo:
+    """ Class for shared photo comments """
+
+    datetime: datetime.datetime
+    user: str
+    ismine: bool
+    text: str
+
+    def asdict(self):
+        return dataclasses.asdict(self)
+
+
+@dataclass
+class LikeInfo:
+    """ Class for shared photo likes """
+
+    datetime: datetime.datetime
+    user: str
+    ismine: bool
+
+    def asdict(self):
+        return dataclasses.asdict(self)
+
+
+# The following methods do not get imported into PhotosDB
+# but will get called by _process_comments
+def _process_comments_4(photosdb):
+    """ process comments and likes info for Photos <= 4 
+        photosdb: PhotosDB instance """
+    raise NotImplementedError(
+        f"Not implemented for database version {photosdb._db_version}."
+    )
+
+
+def _process_comments_5(photosdb):
+    """ process comments and likes info for Photos >= 5 
+        photosdb: PhotosDB instance """
+
+    db = photosdb._tmp_db
+
+    asset_table = _DB_TABLE_NAMES[photosdb._photos_ver]["ASSET"]
+
+    (conn, cursor) = _open_sql_file(db)
+
+    results = conn.execute(
+        """
+        SELECT DISTINCT
+        ZINVITEEHASHEDPERSONID,
+        ZINVITEEFIRSTNAME,
+        ZINVITEELASTNAME,
+        ZINVITEEFULLNAME
+        FROM
+        ZCLOUDSHAREDALBUMINVITATIONRECORD
+        """
+    )
+
+    # order of results
+    # 0: ZINVITEEHASHEDPERSONID,
+    # 1: ZINVITEEFIRSTNAME,
+    # 2: ZINVITEELASTNAME,
+    # 3: ZINVITEEFULLNAME
+
+    photosdb._db_hashed_person_id = {}
+    for row in results.fetchall():
+        person_id = row[0]
+        photosdb._db_hashed_person_id[person_id] = {
+            "first_name": normalize_unicode(row[1]),
+            "last_name": normalize_unicode(row[2]),
+            "full_name": normalize_unicode(row[3]),
+        }
+
+    results = conn.execute(
+        f"""
+        SELECT 
+        {asset_table}.ZUUID, -- UUID of the photo
+        ZCLOUDSHAREDCOMMENT.ZISLIKE, -- comment is actually a "like"
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTDATE, -- date of comment
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTTEXT, -- text of comment
+        ZCLOUDSHAREDCOMMENT.ZCOMMENTERHASHEDPERSONID, -- hashed ID of person who made comment/like
+        ZCLOUDSHAREDCOMMENT.ZISMYCOMMENT -- is my (this user's) comment
+        FROM ZCLOUDSHAREDCOMMENT
+        JOIN {asset_table} ON
+        {asset_table}.Z_PK = ZCLOUDSHAREDCOMMENT.ZCOMMENTEDASSET
+        OR
+        {asset_table}.Z_PK = ZCLOUDSHAREDCOMMENT.ZLIKEDASSET
+    """
+    )
+
+    # order of results
+    # 0: ZGENERICASSET.ZUUID, -- UUID of the photo
+    # 1: ZCLOUDSHAREDCOMMENT.ZISLIKE, -- comment is actually a "like"
+    # 2: ZCLOUDSHAREDCOMMENT.ZCOMMENTDATE, -- date of comment
+    # 3: ZCLOUDSHAREDCOMMENT.ZCOMMENTTEXT, -- text of comment
+    # 4: ZCLOUDSHAREDCOMMENT.ZCOMMENTERHASHEDPERSONID, -- hashed ID of person who made comment/like
+    # 5: ZCLOUDSHAREDCOMMENT.ZISMYCOMMENT -- is my (this user's) comment
+
+    photosdb._db_comments_uuid = {}
+    for row in results:
+        uuid = row[0]
+        is_like = bool(row[1])
+        text = normalize_unicode(row[3])
+        try:
+            user_name = photosdb._db_hashed_person_id[row[4]]["full_name"]
+        except KeyError:
+            user_name = None
+
+        try:
+            dt = datetime.datetime.fromtimestamp(row[2] + TIME_DELTA)
+        except:
+            dt = datetime.datetime(1970, 1, 1)
+
+        ismine = bool(row[5])
+
+        try:
+            db_comments = photosdb._db_comments_uuid[uuid]
+        except KeyError:
+            photosdb._db_comments_uuid[uuid] = {"likes": [], "comments": []}
+            db_comments = photosdb._db_comments_uuid[uuid]
+
+        if is_like:
+            db_comments["likes"].append(LikeInfo(dt, user_name, ismine))
+        elif text:
+            db_comments["comments"].append(CommentInfo(dt, user_name, ismine, text))
+
+    # sort results
+    for uuid in photosdb._db_comments_uuid:
+        if photosdb._db_comments_uuid[uuid]["likes"]:
+            photosdb._db_comments_uuid[uuid]["likes"].sort(key=lambda x: x.datetime)
+        if photosdb._db_comments_uuid[uuid]["comments"]:
+            photosdb._db_comments_uuid[uuid]["comments"].sort(key=lambda x: x.datetime)
+
+    conn.close()

osxphotos/photosdb/photosdb.py

@@ -44,6 +44,7 @@ from ..utils import (
     _get_os_version,
     _open_sql_file,
     get_last_library_path,
+    noop,
     normalize_unicode,
 )
 from .photosdb_utils import get_db_model_version, get_db_version
@@ -67,12 +68,19 @@ class PhotosDB:
         labels_normalized_as_dict,
     )
     from ._photosdb_process_scoreinfo import _process_scoreinfo
+    from ._photosdb_process_comments import _process_comments
 
-    def __init__(self, *dbfile_, dbfile=None):
+    def __init__(self, dbfile=None, verbose=None):
-        """ create a new PhotosDB object 
+        """ Create a new PhotosDB object.
-            path to photos library or database may be specified EITHER as first argument or as named argument dbfile=path 
+
-            specify full path to photos library or photos.db as first argument 
+        Args:
-            specify path to photos library or photos.db using named argument dbfile=path """
+            dbfile: specify full path to photos library or photos.db; if None, will attempt to locate last library opened by Photos.
+            verbose: optional callable function to use for printing verbose text during processing; if None (default), does not print output.
+        
+        Raises:
+            FileNotFoundError if dbfile is not a valid Photos library.
+            TypeError if verbose is not None and not callable.
+        """
 
         # Check OS version
         system = platform.system()
@@ -84,6 +92,12 @@ class PhotosDB:
                 f"you have {system}, OS version: {major}"
             )
 
+        if verbose is None:
+            verbose = noop
+        elif not callable(verbose):
+            raise TypeError("verbose must be callable")
+        self._verbose = verbose
+
         # create a temporary directory
         # tempfile.TemporaryDirectory gets cleaned up when the object does
         self._tempdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
@@ -216,25 +230,7 @@ class PhotosDB:
         if _debug():
             logging.debug(f"dbfile = {dbfile}")
 
-        # get the path to photos library database
+        if dbfile is None:
-        if dbfile_:
-            # got a library path as argument
-            if dbfile:
-                # shouldn't pass via both *args and dbfile=
-                raise TypeError(
-                    f"photos database path must be specified as argument or "
-                    f"named parameter dbfile but not both: args: {dbfile_}, dbfile: {dbfile}",
-                    dbfile_,
-                    dbfile,
-                )
-            elif len(dbfile_) == 1:
-                dbfile = dbfile_[0]
-            else:
-                raise TypeError(
-                    f"__init__ takes only a single argument (photos database path): {dbfile_}",
-                    dbfile_,
-                )
-        elif dbfile is None:
             dbfile = get_last_library_path()
             if dbfile is None:
                 # get_last_library_path must have failed to find library
@@ -262,11 +258,14 @@ class PhotosDB:
         # or photosanalysisd
         self._dbfile = self._dbfile_actual = self._tmp_db = os.path.abspath(dbfile)
 
+        verbose(f"Processing database {self._dbfile}")
+
         # if database is exclusively locked, make a copy of it and use the copy
         # Photos maintains an exclusive lock on the database file while Photos is open
         # photoanalysisd sometimes maintains this lock even after Photos is closed
         # In those cases, make a temp copy of the file for sqlite3 to read
         if _db_is_locked(self._dbfile):
+            verbose(f"Database locked, creating temporary copy.")
             self._tmp_db = self._copy_db_file(self._dbfile)
 
         self._db_version = get_db_version(self._tmp_db)
@@ -279,8 +278,10 @@ class PhotosDB:
                 raise FileNotFoundError(f"dbfile {dbfile} does not exist", dbfile)
             else:
                 self._dbfile_actual = self._tmp_db = dbfile
+                verbose(f"Processing database {self._dbfile_actual}")
                 # if database is exclusively locked, make a copy of it and use the copy
                 if _db_is_locked(self._dbfile_actual):
+                    verbose(f"Database locked, creating temporary copy.")
                     self._tmp_db = self._copy_db_file(self._dbfile_actual)
 
             if _debug():
@@ -549,10 +550,15 @@ class PhotosDB:
         """ process the Photos database to extract info
             works on Photos version <= 4.0 """
 
+        verbose = self._verbose
+        verbose("Processing database.")
+        verbose(f"Database version: {self._db_version}.")
+
         (conn, c) = _open_sql_file(self._tmp_db)
 
         # get info to associate persons with photos
         # then get detected faces in each photo and link to persons
+        verbose("Processing persons in photos.")
         c.execute(
             """ SELECT
                 RKPerson.modelID,
@@ -618,6 +624,7 @@ class PhotosDB:
                 logging.debug(f"Unexpected KeyError _dbpersons_pk[{pk}]")
 
         # get information on detected faces
+        verbose("Processing detected faces in photos.")
         c.execute(
             """ SELECT 
                 RKPerson.modelID,
@@ -655,6 +662,7 @@ class PhotosDB:
             logging.debug(pformat(self._dbfaces_uuid))
 
         # Get info on albums
+        verbose("Processing albums.")
         c.execute(
             """ SELECT 
                 RKAlbum.uuid, 
@@ -797,6 +805,7 @@ class PhotosDB:
             logging.debug(pformat(self._dbfolder_details))
 
         # Get info on keywords
+        verbose("Processing keywords.")
         c.execute(
             """ SELECT
                 RKKeyword.name, 
@@ -824,6 +833,7 @@ class PhotosDB:
             self._dbvolumes[vol[0]] = vol[1]
 
         # Get photo details
+        verbose("Processing photo details.")
         if self._db_version < _PHOTOS_3_VERSION:
             # Photos < 3.0 doesn't have RKVersion.selfPortrait (selfie)
             c.execute(
@@ -846,7 +856,8 @@ class PhotosDB:
                     RKMaster.height,
                     RKMaster.width, 
                     RKMaster.orientation,
-                    RKMaster.fileSize
+                    RKMaster.fileSize,
+                    RKVersion.subType
                     FROM RKVersion, RKMaster
                     WHERE RKVersion.masterUuid = RKMaster.uuid"""
             )
@@ -873,7 +884,8 @@ class PhotosDB:
                     RKMaster.height,
                     RKMaster.width, 
                     RKMaster.orientation,
-                    RKMaster.originalFileSize
+                    RKMaster.originalFileSize,
+                    RKVersion.subType
                     FROM RKVersion, RKMaster 
                     WHERE RKVersion.masterUuid = RKMaster.uuid"""
             )
@@ -919,6 +931,7 @@ class PhotosDB:
         # 37    RKMaster.width,
         # 38    RKMaster.orientation,
         # 39    RKMaster.originalFileSize
+        # 40    RKVersion.subType
 
         for row in c:
             uuid = row[0]
@@ -989,6 +1002,13 @@ class PhotosDB:
 
             self._dbphotos[uuid]["UTI"] = row[22]
 
+            # The UTI in RKMaster will always be UTI of the original
+            # Unlike Photos 5 which changes the UTI to match latest edit
+            self._dbphotos[uuid]["UTI_original"] = row[22]
+
+            # UTI edited will be read from RKModelResource
+            self._dbphotos[uuid]["UTI_edited"] = None
+
             # handle burst photos
             # if burst photo, determine whether or not it's a selected burst photo
             self._dbphotos[uuid]["burstUUID"] = row[23]
@@ -1055,11 +1075,6 @@ class PhotosDB:
             self._dbphotos[uuid]["cloudAvailable"] = None
             self._dbphotos[uuid]["incloud"] = None
 
-            # TODO: NOT YET USED -- PLACEHOLDER for RAW processing (currently only in _process_database5)
-            # original resource choice (e.g. RAW or jpeg)
-            self._dbphotos[uuid]["original_resource_choice"] = None
-            self._dbphotos[uuid]["raw_is_original"] = None
-
             # associated RAW image info
             self._dbphotos[uuid]["has_raw"] = True if row[25] == 7 else False
             self._dbphotos[uuid]["UTI_raw"] = None
@@ -1071,6 +1086,25 @@ class PhotosDB:
             self._dbphotos[uuid]["non_raw_master_uuid"] = row[30]
             self._dbphotos[uuid]["alt_master_uuid"] = row[31]
 
+            # original resource choice (e.g. RAW or jpeg)
+            # In Photos 5+, original_resource_choice set from:
+            # ZADDITIONALASSETATTRIBUTES.ZORIGINALRESOURCECHOICE
+            # = 0 if jpeg is selected as "original" in Photos (the default)
+            # = 1 if RAW is selected as "original" in Photos
+            # RKVersion.subType, RAW always appears to be 16
+            #   4 = mov
+            #   16 = RAW
+            #   32 = JPEG
+            #   64 = TIFF
+            #   2048 = PNG
+            #   32768 = HIEC
+            self._dbphotos[uuid]["original_resource_choice"] = (
+                1 if row[40] == 16 and self._dbphotos[uuid]["has_raw"] else 0
+            )
+            self._dbphotos[uuid]["raw_is_original"] = bool(
+                self._dbphotos[uuid]["original_resource_choice"]
+            )
+
             # recently deleted items
             self._dbphotos[uuid]["intrash"] = True if row[32] == 1 else False
 
@@ -1089,6 +1123,7 @@ class PhotosDB:
             self._dbphotos[uuid]["fok_import_session"] = None
 
         # get additional details from RKMaster, needed for RAW processing
+        verbose("Processing additional photo details.")
         c.execute(
             """ SELECT 
                 RKMaster.uuid,
@@ -1100,7 +1135,8 @@ class PhotosDB:
                 RKMaster.modelID, 
                 RKMaster.fileSize, 
                 RKMaster.isTrulyRaw,
-                RKMaster.alternateMasterUuid
+                RKMaster.alternateMasterUuid,
+                RKMaster.filename
                 FROM RKMaster
             """
         )
@@ -1116,6 +1152,7 @@ class PhotosDB:
         # 7     RKMaster.fileSize,
         # 8     RKMaster.isTrulyRaw,
         # 9     RKMaster.alternateMasterUuid
+        # 10    RKMaster.filename
 
         for row in c:
             uuid = row[0]
@@ -1130,6 +1167,7 @@ class PhotosDB:
             info["fileSize"] = row[7]
             info["isTrulyRAW"] = row[8]
             info["alternateMasterUuid"] = row[9]
+            info["filename"] = row[10]
             self._dbphotos_master[uuid] = info
 
         # get details needed to find path of the edited photos
@@ -1159,7 +1197,6 @@ class PhotosDB:
                     if (
                         row[1] != "UNADJUSTEDNONRAW"
                         and row[1] != "UNADJUSTED"
-                        # and row[4] == "public.jpeg"
                         and row[6] == 2
                     ):
                         if "edit_resource_id" in self._dbphotos[uuid]:
@@ -1173,6 +1210,7 @@ class PhotosDB:
                         # should we return all edits or just most recent one?
                         # For now, return most recent edit
                         self._dbphotos[uuid]["edit_resource_id"] = row[2]
+                        self._dbphotos[uuid]["UTI_edited"] = row[4]
 
         # get details on external edits
         c.execute(
@@ -1245,7 +1283,7 @@ class PhotosDB:
         )
 
         # Order of results
-        # 0  RKMaster.uuid,
+        # 0  RKVersion.uuid,
         # 1  RKMaster.cloudLibraryState,
         # 2  RKCloudResource.available,
         # 3  RKCloudResource.status
@@ -1259,6 +1297,7 @@ class PhotosDB:
                 self._dbphotos[uuid]["incloud"] = True if row[2] == 1 else False
 
         # get location data
+        verbose("Processing location data.")
         # get the country codes
         country_codes = c.execute(
             "SELECT modelID, countryCode "
@@ -1318,22 +1357,34 @@ class PhotosDB:
 
         # add volume name to _dbphotos_master
         for info in self._dbphotos_master.values():
-            info["volume"] = (
+            # issue 230: have seen bad volumeID values
-                self._dbvolumes[info["volumeId"]]
+            try:
-                if info["volumeId"] is not None
+                info["volume"] = (
-                else None
+                    self._dbvolumes[info["volumeId"]]
-            )
+                    if info["volumeId"] is not None
+                    else None
+                )
+            except KeyError:
+                info["volume"] = None
 
         # add data on RAW images
         for info in self._dbphotos.values():
             if info["has_raw"]:
                 raw_uuid = info["raw_master_uuid"]
                 info["raw_info"] = self._dbphotos_master[raw_uuid]
+                info["UTI_raw"] = self._dbphotos_master[raw_uuid]["UTI"]
+                non_raw_uuid = info["non_raw_master_uuid"]
+                info["raw_pair_info"] = self._dbphotos_master[non_raw_uuid]
+            else:
+                info["raw_info"] = None
+                info["UTI_raw"] = None
+                info["raw_pair_info"] = None
 
         # done with the database connection
         conn.close()
 
         # process faces
+        verbose("Processing face details.")
         self._process_faceinfo()
 
         # add faces and keywords to photo data
@@ -1359,13 +1410,18 @@ class PhotosDB:
                 self._dbphotos[uuid]["hasAlbums"] = 0
 
             if self._dbphotos[uuid]["volumeId"] is not None:
-                self._dbphotos[uuid]["volume"] = self._dbvolumes[
+                # issue 230: have seen bad volumeID values
-                    self._dbphotos[uuid]["volumeId"]
+                try:
-                ]
+                    self._dbphotos[uuid]["volume"] = self._dbvolumes[
+                        self._dbphotos[uuid]["volumeId"]
+                    ]
+                except KeyError:
+                    self._dbphotos[uuid]["volume"] = None
             else:
                 self._dbphotos[uuid]["volume"] = None
 
         # done processing, dump debug data if requested
+        verbose("Done processing details from Photos library.")
         if _debug():
             logging.debug("Faces (_dbfaces_uuid):")
             logging.debug(pformat(self._dbfaces_uuid))
@@ -1441,12 +1497,14 @@ class PhotosDB:
 
         if _debug():
             logging.debug(f"_process_database5")
-
+        verbose = self._verbose
+        verbose(f"Processing database.")
         (conn, c) = _open_sql_file(self._tmp_db)
 
         # some of the tables/columns have different names in different versions of Photos
         photos_ver = get_db_model_version(self._tmp_db)
         self._photos_ver = photos_ver
+        verbose(f"Database version: {self._db_version}, {photos_ver}.")
         asset_table = _DB_TABLE_NAMES[photos_ver]["ASSET"]
         keyword_join = _DB_TABLE_NAMES[photos_ver]["KEYWORD_JOIN"]
         album_join = _DB_TABLE_NAMES[photos_ver]["ALBUM_JOIN"]
@@ -1460,6 +1518,7 @@ class PhotosDB:
 
         # get info to associate persons with photos
         # then get detected faces in each photo and link to persons
+        verbose("Processing persons in photos.")
         c.execute(
             """ SELECT
                 ZPERSON.Z_PK,
@@ -1525,6 +1584,7 @@ class PhotosDB:
                 logging.debug(f"Unexpected KeyError _dbpersons_pk[{pk}]")
 
         # get information on detected faces
+        verbose("Processing detected faces in photos.")
         c.execute(
             f""" SELECT
                 ZPERSON.Z_PK,
@@ -1559,6 +1619,7 @@ class PhotosDB:
             logging.debug(pformat(self._dbfaces_uuid))
 
         # get details about albums
+        verbose("Processing albums.")
         c.execute(
             f""" SELECT 
                 ZGENERICALBUM.ZUUID, 
@@ -1677,6 +1738,7 @@ class PhotosDB:
             logging.debug(pformat(self._dbalbum_folders))
 
         # get details on keywords
+        verbose("Processing keywords.")
         c.execute(
             f"""SELECT ZKEYWORD.ZTITLE, {asset_table}.ZUUID
                 FROM {asset_table} 
@@ -1708,6 +1770,7 @@ class PhotosDB:
             logging.debug(self._dbvolumes)
 
         # get details about photos
+        verbose("Processing photo details.")
         logging.debug(f"Getting information about photos")
         c.execute(
             f"""SELECT {asset_table}.ZUUID, 
@@ -1746,7 +1809,8 @@ class PhotosDB:
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALWIDTH, 
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
                 ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE,
-                {depth_state}
+                {depth_state},
+                {asset_table}.ZADJUSTMENTTIMESTAMP
                 FROM {asset_table} 
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = {asset_table}.Z_PK 
                 ORDER BY {asset_table}.ZUUID  """
@@ -1790,6 +1854,7 @@ class PhotosDB:
         # 34   ZADDITIONALASSETATTRIBUTES.ZORIGINALORIENTATION,
         # 35   ZADDITIONALASSETATTRIBUTES.ZORIGINALFILESIZE
         # 36   ZGENERICASSET.ZDEPTHSTATES / ZASSET.ZDEPTHTYPE
+        # 37   ZGENERICASSET.ZADJUSTMENTTIMESTAMP -- when was photo edited?
 
         for row in c:
             uuid = row[0]
@@ -1803,9 +1868,9 @@ class PhotosDB:
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
             # not accounted for
-            info["lastmodifieddate_timestamp"] = row[4]
+            info["lastmodifieddate_timestamp"] = row[37]
             try:
-                info["lastmodifieddate"] = datetime.fromtimestamp(row[4] + TIME_DELTA)
+                info["lastmodifieddate"] = datetime.fromtimestamp(row[37] + TIME_DELTA)
             except ValueError:
                 info["lastmodifieddate"] = None
             except TypeError:
@@ -1866,6 +1931,7 @@ class PhotosDB:
                 info["type"] = None
 
             info["UTI"] = row[18]
+            info["UTI_original"] = None  # filled in later
 
             # handle burst photos
             # if burst photo, determine whether or not it's a selected burst photo
@@ -1997,6 +2063,7 @@ class PhotosDB:
         # 1    ZGENERICASSET.ZIMPORTSESSION
         # 2    ZGENERICASSET.Z_FOK_IMPORTSESSION
         # 3    ZGENERICALBUM.ZUUID,
+        verbose("Processing import sessions.")
         c.execute(
             f"""SELECT
                 {asset_table}.ZUUID,
@@ -2019,6 +2086,7 @@ class PhotosDB:
                 logging.debug(f"No info record for uuid {uuid} for import session")
 
         # Get extended description
+        verbose("Processing additional photo details.")
         c.execute(
             f"""SELECT {asset_table}.ZUUID, 
                 ZASSETDESCRIPTION.ZLONGDESCRIPTION 
@@ -2062,36 +2130,43 @@ class PhotosDB:
         # determine if a photo is missing in Photos 5
 
         # Get info on remote/local availability for photos in shared albums
+        # Also get UTI of original image (zdatastoresubtype = 1)
         c.execute(
             f""" SELECT 
                 {asset_table}.ZUUID, 
                 ZINTERNALRESOURCE.ZLOCALAVAILABILITY, 
-                ZINTERNALRESOURCE.ZREMOTEAVAILABILITY
+                ZINTERNALRESOURCE.ZREMOTEAVAILABILITY,
+                ZINTERNALRESOURCE.ZDATASTORESUBTYPE,
+                ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER,
+                ZUNIFORMTYPEIDENTIFIER.ZIDENTIFIER
                 FROM {asset_table}
                 JOIN ZADDITIONALASSETATTRIBUTES ON ZADDITIONALASSETATTRIBUTES.ZASSET = {asset_table}.Z_PK 
                 JOIN ZINTERNALRESOURCE ON ZINTERNALRESOURCE.ZASSET = ZADDITIONALASSETATTRIBUTES.ZASSET 
+                JOIN ZUNIFORMTYPEIDENTIFIER ON ZUNIFORMTYPEIDENTIFIER.Z_PK = ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER 
                 WHERE  ZDATASTORESUBTYPE = 1 OR ZDATASTORESUBTYPE = 3 """
         )
 
+        # Order of results:
+        # 0 {asset_table}.ZUUID,
+        # 1 ZINTERNALRESOURCE.ZLOCALAVAILABILITY,
+        # 2 ZINTERNALRESOURCE.ZREMOTEAVAILABILITY,
+        # 3 ZINTERNALRESOURCE.ZDATASTORESUBTYPE,
+        # 4 ZINTERNALRESOURCE.ZUNIFORMTYPEIDENTIFIER,
+        # 5 ZUNIFORMTYPEIDENTIFIER.ZIDENTIFIER
+
         for row in c:
             uuid = row[0]
             if uuid in self._dbphotos:
-                #  and self._dbphotos[uuid]["isMissing"] is None:
                 self._dbphotos[uuid]["localAvailability"] = row[1]
                 self._dbphotos[uuid]["remoteAvailability"] = row[2]
-
+                if row[3] == 1:
-                # old = self._dbphotos[uuid]["isMissing"]
+                    self._dbphotos[uuid]["UTI_original"] = row[5]
 
                 if row[1] != 1:
                     self._dbphotos[uuid]["isMissing"] = 1
                 else:
                     self._dbphotos[uuid]["isMissing"] = 0
 
-                # if old is not None and old != self._dbphotos[uuid]["isMissing"]:
-                #     logging.warning(
-                #         f"{uuid} isMissing changed: {old} {self._dbphotos[uuid]['isMissing']}"
-                #     )
-
         # get information on local/remote availability
         c.execute(
             f""" SELECT {asset_table}.ZUUID,
@@ -2108,18 +2183,11 @@ class PhotosDB:
                 self._dbphotos[uuid]["localAvailability"] = row[1]
                 self._dbphotos[uuid]["remoteAvailability"] = row[2]
 
-                # old = self._dbphotos[uuid]["isMissing"]
-
                 if row[1] != 1:
                     self._dbphotos[uuid]["isMissing"] = 1
                 else:
                     self._dbphotos[uuid]["isMissing"] = 0
 
-                # if old is not None and old != self._dbphotos[uuid]["isMissing"]:
-                #     logging.warning(
-                #         f"{uuid} isMissing changed: {old} {self._dbphotos[uuid]['isMissing']}"
-                #     )
-
         # get information about cloud sync state
         c.execute(
             f""" SELECT
@@ -2198,18 +2266,27 @@ class PhotosDB:
         conn.close()
 
         # process face info
+        verbose("Processing face details.")
         self._process_faceinfo()
 
         # process search info
+        verbose("Processing photo labels.")
         self._process_searchinfo()
 
         # process exif info
+        verbose("Processing EXIF details.")
         self._process_exifinfo()
 
         # process computed scores
+        verbose("Processing computed aesthetic scores.")
         self._process_scoreinfo()
 
+        # process shared comments/likes
+        verbose("Processing comments and likes for shared photos.")
+        self._process_comments()
+
         # done processing, dump debug data if requested
+        verbose("Done processing details from Photos library.")
         if _debug():
             logging.debug("Faces (_dbfaces_uuid):")
             logging.debug(pformat(self._dbfaces_uuid))

osxphotos/phototemplate.py

@@ -12,11 +12,13 @@
 import datetime
 import locale
 import os
-import re
 import pathlib
+import re
+from functools import partial
 
 from ._constants import _UNKNOWN_PERSON
 from .datetime_formatter import DateTimeFormatter
+from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 
 # ensure locale set to user's locale
 locale.setlocale(locale.LC_ALL, "")
@@ -28,33 +30,34 @@ TEMPLATE_SUBSTITUTIONS = {
     "{title}": "Title of the photo",
     "{descr}": "Description of the photo",
     "{created.date}": "Photo's creation date in ISO format, e.g. '2020-03-22'",
-    "{created.year}": "4-digit year of file creation time",
+    "{created.year}": "4-digit year of photo creation time",
-    "{created.yy}": "2-digit year of file creation time",
+    "{created.yy}": "2-digit year of photo creation time",
-    "{created.mm}": "2-digit month of the file creation time (zero padded)",
+    "{created.mm}": "2-digit month of the photo creation time (zero padded)",
-    "{created.month}": "Month name in user's locale of the file creation time",
+    "{created.month}": "Month name in user's locale of the photo creation time",
-    "{created.mon}": "Month abbreviation in the user's locale of the file creation time",
+    "{created.mon}": "Month abbreviation in the user's locale of the photo creation time",
-    "{created.dd}": "2-digit day of the month (zero padded) of file creation time",
+    "{created.dd}": "2-digit day of the month (zero padded) of photo creation time",
-    "{created.dow}": "Day of week in user's locale of the file creation time",
+    "{created.dow}": "Day of week in user's locale of the photo creation time",
-    "{created.doy}": "3-digit day of year (e.g Julian day) of file creation time, starting from 1 (zero padded)",
+    "{created.doy}": "3-digit day of year (e.g Julian day) of photo creation time, starting from 1 (zero padded)",
-    "{created.hour}": "2-digit hour of the file creation time",
+    "{created.hour}": "2-digit hour of the photo creation time",
-    "{created.min}": "2-digit minute of the file creation time",
+    "{created.min}": "2-digit minute of the photo creation time",
-    "{created.sec}": "2-digit second of the file creation time",
+    "{created.sec}": "2-digit second of the photo creation time",
     "{created.strftime}": "Apply strftime template to file creation date/time. Should be used in form "
     + "{created.strftime,TEMPLATE} where TEMPLATE is a valid strftime template, e.g. "
     + "{created.strftime,%Y-%U} would result in year-week number of year: '2020-23'. "
     + "If used with no template will return null value. "
     + "See https://strftime.org/ for help on strftime templates.",
     "{modified.date}": "Photo's modification date in ISO format, e.g. '2020-03-22'",
-    "{modified.year}": "4-digit year of file modification time",
+    "{modified.year}": "4-digit year of photo modification time",
-    "{modified.yy}": "2-digit year of file modification time",
+    "{modified.yy}": "2-digit year of photo modification time",
-    "{modified.mm}": "2-digit month of the file modification time (zero padded)",
+    "{modified.mm}": "2-digit month of the photo modification time (zero padded)",
-    "{modified.month}": "Month name in user's locale of the file modification time",
+    "{modified.month}": "Month name in user's locale of the photo modification time",
-    "{modified.mon}": "Month abbreviation in the user's locale of the file modification time",
+    "{modified.mon}": "Month abbreviation in the user's locale of the photo modification time",
-    "{modified.dd}": "2-digit day of the month (zero padded) of the file modification time",
+    "{modified.dd}": "2-digit day of the month (zero padded) of the photo modification time",
-    "{modified.doy}": "3-digit day of year (e.g Julian day) of file modification time, starting from 1 (zero padded)",
+    "{modified.dow}": "Day of week in user's locale of the photo modification time",
-    "{modified.hour}": "2-digit hour of the file modification time",
+    "{modified.doy}": "3-digit day of year (e.g Julian day) of photo modification time, starting from 1 (zero padded)",
-    "{modified.min}": "2-digit minute of the file modification time",
+    "{modified.hour}": "2-digit hour of the photo modification time",
-    "{modified.sec}": "2-digit second of the file modification time",
+    "{modified.min}": "2-digit minute of the photo modification time",
+    "{modified.sec}": "2-digit second of the photo modification time",
     # "{modified.strftime}": "Apply strftime template to file modification date/time. Should be used in form "
     # + "{modified.strftime,TEMPLATE} where TEMPLATE is a valid strftime template, e.g. "
     # + "{modified.strftime,%Y-%U} would result in year-week number of year: '2020-23'. "
@@ -100,6 +103,7 @@ TEMPLATE_SUBSTITUTIONS_MULTI_VALUED = {
     "{person}": "Person(s) / face(s) in a photo",
     "{label}": "Image categorization label associated with a photo (Photos 5 only)",
     "{label_normalized}": "All lower case version of 'label' (Photos 5 only)",
+    "{comment}": "Comment(s) on shared Photos; format is 'Person name: comment text' (Photos 5 only)",
 }
 
 # Just the multi-valued substitution names without the braces
@@ -131,6 +135,9 @@ class PhotoTemplate:
         path_sep=None,
         expand_inplace=False,
         inplace_sep=None,
+        filename=False,
+        dirname=False,
+        replacement=":",
     ):
         """ Render a filename or directory template 
 
@@ -142,6 +149,9 @@ class PhotoTemplate:
                 instead of returning individual strings
             inplace_sep: optional string to use as separator between multi-valued keywords
             with expand_inplace; default is ','
+            filename: if True, template output will be sanitized to produce valid file name
+            dirname: if True, template output will be sanitized to produce valid directory name 
+            replacement: str, value to replace any illegal file path characters with; default = ":"
 
         Returns:
             ([rendered_strings], [unmatched]): tuple of list of rendered strings and list of unmatched template values
@@ -170,13 +180,21 @@ class PhotoTemplate:
         if type(template) is not str:
             raise TypeError(f"template must be type str, not {type(template)}")
 
-        def make_subst_function(self, none_str, get_func=self.get_template_value):
+        # used by make_subst_function to get the value for a template substitution
+        get_func = partial(
+            self.get_template_value,
+            filename=filename,
+            dirname=dirname,
+            replacement=replacement,
+        )
+
+        def make_subst_function(self, none_str, get_func=get_func):
             """ returns: substitution function for use in re.sub 
                 none_str: value to use if substitution lookup is None and no default provided
                 get_func: function that gets the substitution value for a given template field
                         default is get_template_value which handles the single-value fields """
 
-            # closure to capture photo, none_str in subst
+            # closure to capture photo, none_str, filename, dirname in subst
             def subst(matchobj):
                 groups = len(matchobj.groups())
                 if groups == 4:
@@ -186,13 +204,13 @@ class PhotoTemplate:
                         return matchobj.group(0)
 
                     if val is None:
-                        return (
+                        val = (
                             matchobj.group(3)
                             if matchobj.group(3) is not None
                             else none_str
                         )
-                    else:
+
-                        return val
+                    return val
                 else:
                     raise ValueError(
                         f"Unexpected number of groups: expected 4, got {groups}"
@@ -228,18 +246,24 @@ class PhotoTemplate:
         #    '2011/Album2/keyword1/person1',
         #    '2011/Album2/keyword2/person1',]
 
-        rendered_strings = set([rendered])
+        rendered_strings = [rendered]
         for field in MULTI_VALUE_SUBSTITUTIONS:
             # Build a regex that matches only the field being processed
             re_str = r"(?<!\\)\{(" + field + r")(,(([\w\-\%. ]{0,})))?\}"
             regex_multi = re.compile(re_str)
 
-            # holds each of the new rendered_strings, set() to avoid duplicates
+            # holds each of the new rendered_strings, dict to avoid repeats (dict.keys())
-            new_strings = set()
+            new_strings = {}
 
             for str_template in rendered_strings:
                 if regex_multi.search(str_template):
-                    values = self.get_template_value_multi(field, path_sep)
+                    values = self.get_template_value_multi(
+                        field,
+                        path_sep,
+                        filename=filename,
+                        dirname=dirname,
+                        replacement=replacement,
+                    )
                     if expand_inplace:
                         # instead of returning multiple strings, join values into a single string
                         val = (
@@ -248,11 +272,11 @@ class PhotoTemplate:
                             else None
                         )
 
-                        def lookup_template_value_multi(lookup_value, default):
+                        def lookup_template_value_multi(lookup_value, _):
                             """ Closure passed to make_subst_function get_func 
                                     Capture val and field in the closure 
                                     Allows make_subst_function to be re-used w/o modification
-                                    default is not used but required so signature matches get_template_value """
+                                    _ is not used but required so signature matches get_template_value """
                             if lookup_value == field:
                                 return val
                             else:
@@ -269,11 +293,11 @@ class PhotoTemplate:
                         # create a new template string for each value
                         for val in values:
 
-                            def lookup_template_value_multi(lookup_value, default):
+                            def lookup_template_value_multi(lookup_value, _):
                                 """ Closure passed to make_subst_function get_func 
                                     Capture val and field in the closure 
                                     Allows make_subst_function to be re-used w/o modification
-                                    default is not used but required so signature matches get_template_value """
+                                    _ is not used but required so signature matches get_template_value """
                                 if lookup_value == field:
                                     return val
                                 else:
@@ -285,10 +309,10 @@ class PhotoTemplate:
                                 self, none_str, get_func=lookup_template_value_multi
                             )
                             new_string = regex_multi.sub(subst, str_template)
-                            new_strings.add(new_string)
+                            new_strings[new_string] = 1
 
                         # update rendered_strings for the next field to process
-                        rendered_strings = new_strings
+                        rendered_strings = list(new_strings.keys())
 
         # find any {fields} that weren't replaced
         unmatched = []
@@ -307,14 +331,24 @@ class PhotoTemplate:
             for rendered_str in rendered_strings
         ]
 
+        if filename:
+            rendered_strings = [
+                sanitize_filename(rendered_str) for rendered_str in rendered_strings
+            ]
+
         return rendered_strings, unmatched
 
-    def get_template_value(self, field, default):
+    def get_template_value(
+        self, field, default, filename=False, dirname=False, replacement=":"
+    ):
         """lookup value for template field (single-value template substitutions)
 
         Args:
             field: template field to find value for.
             default: the default value provided by the user
+            filename: if True, template output will be sanitized to produce valid file name
+            dirname: if True, template output will be sanitized to produce valid directory name 
+            replacement: str, value to replace any illegal file path characters with; default = ":"
         
         Returns:
             The matching template value (which may be None).
@@ -327,289 +361,242 @@ class PhotoTemplate:
         if self.today is None:
             self.today = datetime.datetime.now()
 
-        # must be a valid keyword
+        value = None
+
+        # wouldn't a switch/case statement be nice...
         if field == "name":
-            return pathlib.Path(self.photo.filename).stem
+            value = pathlib.Path(self.photo.filename).stem
-
+        elif field == "original_name":
-        if field == "original_name":
+            value = pathlib.Path(self.photo.original_filename).stem
-            return pathlib.Path(self.photo.original_filename).stem
+        elif field == "title":
-
+            value = self.photo.title
-        if field == "title":
+        elif field == "descr":
-            return self.photo.title
+            value = self.photo.description
-
+        elif field == "created.date":
-        if field == "descr":
+            value = DateTimeFormatter(self.photo.date).date
-            return self.photo.description
+        elif field == "created.year":
-
+            value = DateTimeFormatter(self.photo.date).year
-        if field == "created.date":
+        elif field == "created.yy":
-            return DateTimeFormatter(self.photo.date).date
+            value = DateTimeFormatter(self.photo.date).yy
-
+        elif field == "created.mm":
-        if field == "created.year":
+            value = DateTimeFormatter(self.photo.date).mm
-            return DateTimeFormatter(self.photo.date).year
+        elif field == "created.month":
-
+            value = DateTimeFormatter(self.photo.date).month
-        if field == "created.yy":
+        elif field == "created.mon":
-            return DateTimeFormatter(self.photo.date).yy
+            value = DateTimeFormatter(self.photo.date).mon
-
+        elif field == "created.dd":
-        if field == "created.mm":
+            value = DateTimeFormatter(self.photo.date).dd
-            return DateTimeFormatter(self.photo.date).mm
+        elif field == "created.dow":
-
+            value = DateTimeFormatter(self.photo.date).dow
-        if field == "created.month":
+        elif field == "created.doy":
-            return DateTimeFormatter(self.photo.date).month
+            value = DateTimeFormatter(self.photo.date).doy
-
+        elif field == "created.hour":
-        if field == "created.mon":
+            value = DateTimeFormatter(self.photo.date).hour
-            return DateTimeFormatter(self.photo.date).mon
+        elif field == "created.min":
-
+            value = DateTimeFormatter(self.photo.date).min
-        if field == "created.dd":
+        elif field == "created.sec":
-            return DateTimeFormatter(self.photo.date).dd
+            value = DateTimeFormatter(self.photo.date).sec
-
+        elif field == "created.strftime":
-        if field == "created.dow":
-            return DateTimeFormatter(self.photo.date).dow
-
-        if field == "created.doy":
-            return DateTimeFormatter(self.photo.date).doy
-
-        if field == "created.hour":
-            return DateTimeFormatter(self.photo.date).hour
-
-        if field == "created.min":
-            return DateTimeFormatter(self.photo.date).min
-
-        if field == "created.sec":
-            return DateTimeFormatter(self.photo.date).sec
-
-        if field == "created.strftime":
             if default:
                 try:
-                    return self.photo.date.strftime(default)
+                    value = self.photo.date.strftime(default)
                 except:
                     raise ValueError(f"Invalid strftime template: '{default}'")
             else:
-                return None
+                value = None
-
+        elif field == "modified.date":
-        if field == "modified.date":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).date
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.year":
-        if field == "modified.year":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).year
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.yy":
-        if field == "modified.yy":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).yy
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.mm":
-        if field == "modified.mm":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).mm
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.month":
-        if field == "modified.month":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).month
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.mon":
-        if field == "modified.mon":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).mon
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.dd":
-        if field == "modified.dd":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).dd
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.dow":
-        if field == "modified.doy":
+            value = (
-            return (
+                DateTimeFormatter(self.photo.date_modified).dow
+                if self.photo.date_modified
+                else None
+            )
+        elif field == "modified.doy":
+            value = (
                 DateTimeFormatter(self.photo.date_modified).doy
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.hour":
-        if field == "modified.hour":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).hour
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.min":
-        if field == "modified.min":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).min
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "modified.sec":
-        if field == "modified.sec":
+            value = (
-            return (
                 DateTimeFormatter(self.photo.date_modified).sec
                 if self.photo.date_modified
                 else None
             )
-
+        elif field == "today.date":
-        # TODO: disabling modified.strftime for now because now clean way to pass
+            value = DateTimeFormatter(self.today).date
-        # a default value if modified time is None
+        elif field == "today.year":
-        # if field == "modified.strftime":
+            value = DateTimeFormatter(self.today).year
-        #     if default and self.photo.date_modified:
+        elif field == "today.yy":
-        #         try:
+            value = DateTimeFormatter(self.today).yy
-        #             return self.photo.date_modified.strftime(default)
+        elif field == "today.mm":
-        #         except:
+            value = DateTimeFormatter(self.today).mm
-        #             raise ValueError(f"Invalid strftime template: '{default}'")
+        elif field == "today.month":
-        #     else:
+            value = DateTimeFormatter(self.today).month
-        #         return None
+        elif field == "today.mon":
-
+            value = DateTimeFormatter(self.today).mon
-        if field == "today.date":
+        elif field == "today.dd":
-            return DateTimeFormatter(self.today).date
+            value = DateTimeFormatter(self.today).dd
-
+        elif field == "today.dow":
-        if field == "today.year":
+            value = DateTimeFormatter(self.today).dow
-            return DateTimeFormatter(self.today).year
+        elif field == "today.doy":
-
+            value = DateTimeFormatter(self.today).doy
-        if field == "today.yy":
+        elif field == "today.hour":
-            return DateTimeFormatter(self.today).yy
+            value = DateTimeFormatter(self.today).hour
-
+        elif field == "today.min":
-        if field == "today.mm":
+            value = DateTimeFormatter(self.today).min
-            return DateTimeFormatter(self.today).mm
+        elif field == "today.sec":
-
+            value = DateTimeFormatter(self.today).sec
-        if field == "today.month":
+        elif field == "today.strftime":
-            return DateTimeFormatter(self.today).month
-
-        if field == "today.mon":
-            return DateTimeFormatter(self.today).mon
-
-        if field == "today.dd":
-            return DateTimeFormatter(self.today).dd
-
-        if field == "today.dow":
-            return DateTimeFormatter(self.today).dow
-
-        if field == "today.doy":
-            return DateTimeFormatter(self.today).doy
-
-        if field == "today.hour":
-            return DateTimeFormatter(self.today).hour
-
-        if field == "today.min":
-            return DateTimeFormatter(self.today).min
-
-        if field == "today.sec":
-            return DateTimeFormatter(self.today).sec
-
-        if field == "today.strftime":
             if default:
                 try:
-                    return self.today.strftime(default)
+                    value = self.today.strftime(default)
                 except:
                     raise ValueError(f"Invalid strftime template: '{default}'")
             else:
-                return None
+                value = None
-
+        elif field == "place.name":
-        if field == "place.name":
+            value = self.photo.place.name if self.photo.place else None
-            return self.photo.place.name if self.photo.place else None
+        elif field == "place.country_code":
-
+            value = self.photo.place.country_code if self.photo.place else None
-        if field == "place.country_code":
+        elif field == "place.name.country":
-            return self.photo.place.country_code if self.photo.place else None
+            value = (
-
-        if field == "place.name.country":
-            return (
                 self.photo.place.names.country[0]
                 if self.photo.place and self.photo.place.names.country
                 else None
             )
-
+        elif field == "place.name.state_province":
-        if field == "place.name.state_province":
+            value = (
-            return (
                 self.photo.place.names.state_province[0]
                 if self.photo.place and self.photo.place.names.state_province
                 else None
             )
-
+        elif field == "place.name.city":
-        if field == "place.name.city":
+            value = (
-            return (
                 self.photo.place.names.city[0]
                 if self.photo.place and self.photo.place.names.city
                 else None
             )
-
+        elif field == "place.name.area_of_interest":
-        if field == "place.name.area_of_interest":
+            value = (
-            return (
                 self.photo.place.names.area_of_interest[0]
                 if self.photo.place and self.photo.place.names.area_of_interest
                 else None
             )
-
+        elif field == "place.address":
-        if field == "place.address":
+            value = (
-            return (
                 self.photo.place.address_str
                 if self.photo.place and self.photo.place.address_str
                 else None
             )
-
+        elif field == "place.address.street":
-        if field == "place.address.street":
+            value = (
-            return (
                 self.photo.place.address.street
                 if self.photo.place and self.photo.place.address.street
                 else None
             )
-
+        elif field == "place.address.city":
-        if field == "place.address.city":
+            value = (
-            return (
                 self.photo.place.address.city
                 if self.photo.place and self.photo.place.address.city
                 else None
             )
-
+        elif field == "place.address.state_province":
-        if field == "place.address.state_province":
+            value = (
-            return (
                 self.photo.place.address.state_province
                 if self.photo.place and self.photo.place.address.state_province
                 else None
             )
-
+        elif field == "place.address.postal_code":
-        if field == "place.address.postal_code":
+            value = (
-            return (
                 self.photo.place.address.postal_code
                 if self.photo.place and self.photo.place.address.postal_code
                 else None
             )
-
+        elif field == "place.address.country":
-        if field == "place.address.country":
+            value = (
-            return (
                 self.photo.place.address.country
                 if self.photo.place and self.photo.place.address.country
                 else None
             )
-
+        elif field == "place.address.country_code":
-        if field == "place.address.country_code":
+            value = (
-            return (
                 self.photo.place.address.iso_country_code
                 if self.photo.place and self.photo.place.address.iso_country_code
                 else None
             )
+        else:
+            # if here, didn't get a match
+            raise ValueError(f"Unhandled template value: {field}")
 
-        # if here, didn't get a match
+        if filename:
-        raise ValueError(f"Unhandled template value: {field}")
+            value = sanitize_pathpart(value, replacement=replacement)
+        elif dirname:
+            value = sanitize_dirname(value, replacement=replacement)
+        return value
 
-    def get_template_value_multi(self, field, path_sep):
+    def get_template_value_multi(
+        self, field, path_sep, filename=False, dirname=False, replacement=":"
+    ):
         """lookup value for template field (multi-value template substitutions)
 
         Args:
             field: template field to find value for.
             path_sep: path separator to use for folder_album field
+            dirname: if True, values will be sanitized to be valid directory names; default = False
         
         Returns:
             List of the matching template values or [None].
@@ -621,9 +608,6 @@ class PhotoTemplate:
         """ return list of values for a multi-valued template field """
         if field == "album":
             values = self.photo.albums
-            values = [
-                value.replace("/", ":") for value in values
-            ]  # TODO: temp fix for issue #213
         elif field == "keyword":
             values = self.photo.keywords
         elif field == "person":
@@ -640,17 +624,46 @@ class PhotoTemplate:
             for album in self.photo.album_info:
                 if album.folder_names:
                     # album in folder
-                    folder = path_sep.join(album.folder_names)
+                    if dirname:
-                    folder += path_sep + album.title.replace(
+                        # being used as a filepath so sanitize each part
-                        "/", ":"
+                        folder = path_sep.join(
-                    )  # TODO: temp fix for issue #213
+                            sanitize_dirname(f, replacement=replacement)
+                            for f in album.folder_names
+                        )
+                        folder += path_sep + sanitize_dirname(
+                            album.title, replacement=replacement
+                        )
+                    else:
+                        folder = path_sep.join(album.folder_names)
+                        folder += path_sep + album.title
                     values.append(folder)
                 else:
                     # album not in folder
-                    values.append(album.title.replace("/", ":"))
+                    if dirname:
+                        values.append(
+                            sanitize_dirname(album.title, replacement=replacement)
+                        )
+                    else:
+                        values.append(album.title)
+        elif field == "comment":
+            values = [
+                f"{comment.user}: {comment.text}" for comment in self.photo.comments
+            ]
         else:
-            raise ValueError(f"Unhandleded template value: {field}")
+            raise ValueError(f"Unhandled template value: {field}")
+
+        # sanitize directory names if needed, folder_album handled differently above
+        if filename:
+            values = [
+                sanitize_pathpart(value, replacement=replacement) for value in values
+            ]
+        elif dirname and field != "folder_album":
+            # skip folder_album because it would have been handled above
+            values = [
+                sanitize_dirname(value, replacement=replacement) for value in values
+            ]
 
         # If no values, insert None so code below will substite none_str for None
         values = values or [None]
         return values
+

osxphotos/placeinfo.py

@@ -491,7 +491,7 @@ class PlaceInfo4(PlaceInfo):
         }
         return "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
 
-    def as_dict(self):
+    def asdict(self):
         return {
             "name": self.name,
             "names": self.names._asdict(),
@@ -634,7 +634,7 @@ class PlaceInfo5(PlaceInfo):
         }
         return "PlaceInfo(" + ", ".join([f"{k}='{v}'" for k, v in info.items()]) + ")"
 
-    def as_dict(self):
+    def asdict(self):
         return {
             "name": self.name,
             "names": self.names._asdict(),

osxphotos/utils.py

@@ -57,6 +57,9 @@ def _debug():
     """ returns True if debugging turned on (via _set_debug), otherwise, false """
     return _DEBUG
 
+def noop(*args, **kwargs):
+    """ do nothing (no operation) """
+    pass
 
 def _get_os_version():
     # returns tuple containing OS version

requirements.txt

@@ -202,5 +202,6 @@ virtualenv==20.0.30
 wcwidth==0.1.9
 webencodings==0.5.1
 wrapt==1.11.1
+wurlitzer==2.0.1
 yarl==1.4.2
 zipp==0.5.2

setup.py

@@ -78,6 +78,7 @@ setup(
         "bpylist2==3.0.2",
         "pathvalidate==2.2.1",
         "dataclasses==0.7;python_version<'3.7'",
+        "wurlitzer>=2.0.1",
     ],
     entry_points={"console_scripts": ["osxphotos=osxphotos.__main__:cli"]},
     include_package_data=True,

tests/Test-10.14.6.photoslibrary/private/com.apple.Photos/appPrivateData.plist

@@ -36,7 +36,7 @@
 			<key>other</key>
 			<integer>0</integer>
 			<key>photos</key>
-			<integer>6</integer>
+			<integer>11</integer>
 			<key>videos</key>
 			<integer>0</integer>
 		</dict>

tests/Test-10.14.6.photoslibrary/private/com.apple.photoanalysisd/GraphService/PhotoAnalysisServicePreferences.plist

@@ -3,8 +3,8 @@
 <plist version="1.0">
 <dict>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
-	<date>2020-07-27T03:16:28Z</date>
+	<date>2020-10-09T16:14:42Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2020-07-27T12:35:43Z</date>
+	<date>2020-10-10T05:21:03Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/private/com.apple.photomodel/appPrivateData.plist

@@ -5,7 +5,7 @@
 	<key>LithiumMessageTracer</key>
 	<dict>
 		<key>LastReportedDate</key>
-		<date>2020-07-27T03:18:40Z</date>
+		<date>2020-10-04T23:49:39Z</date>
 	</dict>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/moments/analysismetadata.plist

@@ -11,6 +11,6 @@
 	<key>PLLastRevGeoForcedProviderOutOfDateCheckVersionKey</key>
 	<integer>1</integer>
 	<key>PLLastRevGeoVerFileFetchDateKey</key>
-	<date>2020-07-27T03:16:25Z</date>
+	<date>2020-10-04T23:43:17Z</date>
 </dict>
 </plist>

tests/Test-10.14.6.photoslibrary/resources/moments/historicalmarker.plist

@@ -3,7 +3,7 @@
 <plist version="1.0">
 <dict>
 	<key>LastHistoryRowId</key>
-	<integer>707</integer>
+	<integer>948</integer>
 	<key>LibraryBuildTag</key>
 	<string>D8C4AAA1-3AB6-4A65-BEBD-99CC3E5D433E</string>
 	<key>LibrarySchemaVersion</key>

tests/Test-10.14.6.photoslibrary/resources/recovery/Info.plist

@@ -9,7 +9,7 @@
 	<key>HistoricalMarker</key>
 	<dict>
 		<key>LastHistoryRowId</key>
-		<integer>707</integer>
+		<integer>948</integer>
 		<key>LibraryBuildTag</key>
 		<string>D8C4AAA1-3AB6-4A65-BEBD-99CC3E5D433E</string>
 		<key>LibrarySchemaVersion</key>
@@ -24,7 +24,7 @@
 	<key>SnapshotCompletedDate</key>
 	<date>2019-07-27T13:16:43Z</date>
 	<key>SnapshotLastValidated</key>
-	<date>2020-07-27T03:18:40Z</date>
+	<date>2020-10-10T05:22:36Z</date>
 	<key>SnapshotTables</key>
 	<dict/>
 </dict>

tests/Test-10.15.6.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>9575E48B-8D5F-5654-ABAC-4431B1167324</string>
 	<key>pid</key>
-	<integer>80508</integer>
+	<integer>36387</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.6.photoslibrary/database/search/searchMetadata.plist

@@ -0,0 +1,188 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>BlacklistedMeaningsByMeaning</key>
+	<dict/>
+	<key>MePersonUUID</key>
+	<string>39488755-78C0-40B2-B378-EDA280E1823C</string>
+	<key>SceneWhitelist</key>
+	<array>
+		<string>Graduation</string>
+		<string>Aquarium</string>
+		<string>Food</string>
+		<string>Ice Skating</string>
+		<string>Mountain</string>
+		<string>Cliff</string>
+		<string>Basketball</string>
+		<string>Tennis</string>
+		<string>Jewelry</string>
+		<string>Cheese</string>
+		<string>Softball</string>
+		<string>Football</string>
+		<string>Circus</string>
+		<string>Jet Ski</string>
+		<string>Playground</string>
+		<string>Carousel</string>
+		<string>Paint Ball</string>
+		<string>Windsurfing</string>
+		<string>Sailboat</string>
+		<string>Sunbathing</string>
+		<string>Dam</string>
+		<string>Fireplace</string>
+		<string>Flower</string>
+		<string>Scuba</string>
+		<string>Hiking</string>
+		<string>Cetacean</string>
+		<string>Pier</string>
+		<string>Bowling</string>
+		<string>Snowboarding</string>
+		<string>Zoo</string>
+		<string>Snowmobile</string>
+		<string>Theater</string>
+		<string>Boat</string>
+		<string>Casino</string>
+		<string>Car</string>
+		<string>Diving</string>
+		<string>Cycling</string>
+		<string>Musical Instrument</string>
+		<string>Board Game</string>
+		<string>Castle</string>
+		<string>Sunset Sunrise</string>
+		<string>Martial Arts</string>
+		<string>Motocross</string>
+		<string>Submarine</string>
+		<string>Cat</string>
+		<string>Snow</string>
+		<string>Kiteboarding</string>
+		<string>Squash</string>
+		<string>Geyser</string>
+		<string>Music</string>
+		<string>Archery</string>
+		<string>Desert</string>
+		<string>Blackjack</string>
+		<string>Fireworks</string>
+		<string>Sportscar</string>
+		<string>Feline</string>
+		<string>Soccer</string>
+		<string>Museum</string>
+		<string>Baby</string>
+		<string>Fencing</string>
+		<string>Railroad</string>
+		<string>Nascar</string>
+		<string>Sky Surfing</string>
+		<string>Bird</string>
+		<string>Games</string>
+		<string>Baseball</string>
+		<string>Dressage</string>
+		<string>Snorkeling</string>
+		<string>Pyramid</string>
+		<string>Kite</string>
+		<string>Rowboat</string>
+		<string>Golf</string>
+		<string>Watersports</string>
+		<string>Lightning</string>
+		<string>Canyon</string>
+		<string>Auditorium</string>
+		<string>Night Sky</string>
+		<string>Karaoke</string>
+		<string>Skiing</string>
+		<string>Parade</string>
+		<string>Forest</string>
+		<string>Hot Air Balloon</string>
+		<string>Dragon Parade</string>
+		<string>Easter Egg</string>
+		<string>Monument</string>
+		<string>Jungle</string>
+		<string>Thanksgiving</string>
+		<string>Jockey Horse</string>
+		<string>Stadium</string>
+		<string>Airplane</string>
+		<string>Ballet</string>
+		<string>Yoga</string>
+		<string>Coral Reef</string>
+		<string>Skating</string>
+		<string>Wrestling</string>
+		<string>Bicycle</string>
+		<string>Tattoo</string>
+		<string>Amusement Park</string>
+		<string>Canoe</string>
+		<string>Cheerleading</string>
+		<string>Ping Pong</string>
+		<string>Fishing</string>
+		<string>Magic</string>
+		<string>Reptile</string>
+		<string>Winter Sport</string>
+		<string>Waterfall</string>
+		<string>Train</string>
+		<string>Bonsai</string>
+		<string>Surfing</string>
+		<string>Dog</string>
+		<string>Cake</string>
+		<string>Sledding</string>
+		<string>Sandcastle</string>
+		<string>Glacier</string>
+		<string>Lighthouse</string>
+		<string>Equestrian</string>
+		<string>Rafting</string>
+		<string>Shore</string>
+		<string>Hockey</string>
+		<string>Santa Claus</string>
+		<string>Formula One Car</string>
+		<string>Sport</string>
+		<string>Vehicle</string>
+		<string>Boxing</string>
+		<string>Rollerskating</string>
+		<string>Underwater</string>
+		<string>Orchestra</string>
+		<string>Carnival</string>
+		<string>Rocket</string>
+		<string>Skateboarding</string>
+		<string>Helicopter</string>
+		<string>Performance</string>
+		<string>Oktoberfest</string>
+		<string>Water Polo</string>
+		<string>Skate Park</string>
+		<string>Animal</string>
+		<string>Nightclub</string>
+		<string>String Instrument</string>
+		<string>Dinosaur</string>
+		<string>Gymnastics</string>
+		<string>Cricket</string>
+		<string>Volcano</string>
+		<string>Lake</string>
+		<string>Aurora</string>
+		<string>Dancing</string>
+		<string>Concert</string>
+		<string>Rock Climbing</string>
+		<string>Hang Glider</string>
+		<string>Rodeo</string>
+		<string>Fish</string>
+		<string>Art</string>
+		<string>Motorcycle</string>
+		<string>Volleyball</string>
+		<string>Wake Boarding</string>
+		<string>Badminton</string>
+		<string>Motor Sport</string>
+		<string>Sumo</string>
+		<string>Parasailing</string>
+		<string>Skydiving</string>
+		<string>Kickboxing</string>
+		<string>Pinata</string>
+		<string>Foosball</string>
+		<string>Go Kart</string>
+		<string>Poker</string>
+		<string>Kayak</string>
+		<string>Swimming</string>
+		<string>Atv</string>
+		<string>Beach</string>
+		<string>Dartboard</string>
+		<string>Athletics</string>
+		<string>Camping</string>
+		<string>Tornado</string>
+		<string>Billiards</string>
+		<string>Rugby</string>
+		<string>Airshow</string>
+	</array>
+</dict>
+</plist>