Allow multiple characters as path_sep, #634

* docs: update .all-contributorsrc [skip ci]

* docs: update README.md [skip ci]

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

.all-contributorsrc

@@ -257,7 +257,9 @@
       "avatar_url": "https://avatars.githubusercontent.com/u/21261491?v=4",
       "profile": "https://github.com/oPromessa",
       "contributions": [
-        "bug"
+        "bug",
+        "ideas",
+        "test"
       ]
     },
     {

CHANGELOG.md

@@ -4,6 +4,98 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.45.10](https://github.com/RhetTbull/osxphotos/compare/v0.45.9...v0.45.10)
+
+> 12 February 2022
+
+- Added --force-update, #621 [`30abddd`](https://github.com/RhetTbull/osxphotos/commit/30abdddaf3765f1d604984d4781b78b7806871e1)
+
+#### [v0.45.9](https://github.com/RhetTbull/osxphotos/compare/v0.45.8...v0.45.9)
+
+> 12 February 2022
+
+- Added --force-update, #621 [`bfa888a`](https://github.com/RhetTbull/osxphotos/commit/bfa888adc5658a2845dcaa9b7ea360926ed4f000)
+- Refactored fix for #627 [`5fb686a`](https://github.com/RhetTbull/osxphotos/commit/5fb686ac0c231932c2695fc550a0824307bd3c5f)
+- Fix for #630 [`ac4083b`](https://github.com/RhetTbull/osxphotos/commit/ac4083bfbbabc8550718f0f7f8aadc635c05eb25)
+
+#### [v0.45.8](https://github.com/RhetTbull/osxphotos/compare/v0.45.6...v0.45.8)
+
+> 5 February 2022
+
+- Fixed exiftool to ignore unsupported file types, #615 [`1ae6270`](https://github.com/RhetTbull/osxphotos/commit/1ae627056113fc4655f1b24cfbbdf0efc04489e7)
+- Updated tests [`55a601c`](https://github.com/RhetTbull/osxphotos/commit/55a601c07ea1384623c55d5c1d26b568df5d7823)
+- Additional fix for #615 [`1d6bc4e`](https://github.com/RhetTbull/osxphotos/commit/1d6bc4e09e3c2359a21f842fadd781920606812e)
+
+#### [v0.45.6](https://github.com/RhetTbull/osxphotos/compare/v0.45.5...v0.45.6)
+
+> 5 February 2022
+
+- Fix for unicode in query strings, #618 [`9b247ac`](https://github.com/RhetTbull/osxphotos/commit/9b247acd1cc4b2def59fdd18a6fb3c8eb9914f11)
+- Fix for --name searching only original_filename on Photos 5+, #594 [`cd02144`](https://github.com/RhetTbull/osxphotos/commit/cd02144ac33cc1c13a20358133971c84d35b8a57)
+
+#### [v0.45.5](https://github.com/RhetTbull/osxphotos/compare/v0.45.4...v0.45.5)
+
+> 5 February 2022
+
+- Fix for #561, no really, I mean it this time [`b3d3e14`](https://github.com/RhetTbull/osxphotos/commit/b3d3e14ffe41fbb22edb614b24f3985f379766a2)
+- Updated docs [skip ci] [`2b9ea11`](https://github.com/RhetTbull/osxphotos/commit/2b9ea11701799af9a661a8e2af70fca97235f487)
+- Updated tests for #561 [skip ci] [`77a49a0`](https://github.com/RhetTbull/osxphotos/commit/77a49a09a1bee74113a7114c543fbc25fa410ffc)
+
+#### [v0.45.4](https://github.com/RhetTbull/osxphotos/compare/v0.45.3...v0.45.4)
+
+> 3 February 2022
+
+- docs: add oPromessa as a contributor for ideas, test [`#611`](https://github.com/RhetTbull/osxphotos/pull/611)
+- Fix for filenames with special characters, #561, #618 [`f3063d3`](https://github.com/RhetTbull/osxphotos/commit/f3063d35be3c96342d83dbd87ddd614a2001bff4)
+- Updated docs [skip ci] [`06c5bbf`](https://github.com/RhetTbull/osxphotos/commit/06c5bbfcfdf591a4a5d43f1456adaa27385fe01a)
+- Added progress counter, #601 [`7ab5007`](https://github.com/RhetTbull/osxphotos/commit/7ab500740b28594dcd778140e10991f839220e9d)
+- Updated known issues [skip ci] [`e32090b`](https://github.com/RhetTbull/osxphotos/commit/e32090bf39cb786171b49443f878ffdbab774420)
+
+#### [v0.45.3](https://github.com/RhetTbull/osxphotos/compare/v0.45.2...v0.45.3)
+
+> 29 January 2022
+
+- Added --timestamp option for --verbose, #600 [`d8c2f99`](https://github.com/RhetTbull/osxphotos/commit/d8c2f99c06bc6f72bf2cb1a13c5765824fe3cbba)
+- Updated docs [skip ci] [`5fc2813`](https://github.com/RhetTbull/osxphotos/commit/5fc28139ea0374bc3e228c0432b8a41ada430389)
+- Updated formatting for elapsed time, #604 [`16d3f74`](https://github.com/RhetTbull/osxphotos/commit/16d3f743664396d43b3b3028a5e7a919ec56d9e1)
+
+#### [v0.45.2](https://github.com/RhetTbull/osxphotos/compare/v0.45.0...v0.45.2)
+
+> 29 January 2022
+
+- Implemented #605, refactor out export2 [`235dea3`](https://github.com/RhetTbull/osxphotos/commit/235dea329c98ab8fa61565c09a1b4a83e5d99043)
+- Fix for #564, --preview with --download-missing [`5afdf6f`](https://github.com/RhetTbull/osxphotos/commit/5afdf6fc20a3cb6eb2b0217d8b3be20295eb7ba4)
+
+#### [v0.45.0](https://github.com/RhetTbull/osxphotos/compare/v0.44.13...v0.45.0)
+
+> 28 January 2022
+
+- Performance improvements and refactoring, #462, partial for #591 [`22964af`](https://github.com/RhetTbull/osxphotos/commit/22964afc6988166218413125d7a62348bb858a83)
+- Refactored photoexporter for performance, #591 [`6843b86`](https://github.com/RhetTbull/osxphotos/commit/6843b8661d41d42368794c77304fc07194e7af18)
+- Performance improvements, partial for #591 [`3bc53fd`](https://github.com/RhetTbull/osxphotos/commit/3bc53fd92b3222c6959e7aa12310811db41b83fe)
+
+#### [v0.44.13](https://github.com/RhetTbull/osxphotos/compare/v0.44.12...v0.44.13)
+
+> 24 January 2022
+
+- Removed exportdb requirement from PhotoTemplate [`6af124e`](https://github.com/RhetTbull/osxphotos/commit/6af124e4d3a0e26c48f435452920020cd42afa1c)
+- Version bump [`bd31120`](https://github.com/RhetTbull/osxphotos/commit/bd3112056920806f565be2c0c12caf4f2aff5231)
+
+#### [v0.44.12](https://github.com/RhetTbull/osxphotos/compare/v0.44.11...v0.44.12)
+
+> 23 January 2022
+
+- Added query options to repl, #597 [`7855801`](https://github.com/RhetTbull/osxphotos/commit/785580115b29f5ccb895de22be1243f56dbb43dc)
+- Added run command, #598 [`b4bd04c`](https://github.com/RhetTbull/osxphotos/commit/b4bd04c1461d0b427937f541403305bc979bcf4f)
+- Bug fix for get_photos_library_version [`e88c6b8`](https://github.com/RhetTbull/osxphotos/commit/e88c6b8a59dfd947f6cf3c7eac9c92519ab781a3)
+
+#### [v0.44.11](https://github.com/RhetTbull/osxphotos/compare/v0.44.10...v0.44.11)
+
+> 23 January 2022
+
+- creat unit test for __all__ [`#599`](https://github.com/RhetTbull/osxphotos/pull/599)
+- Performance improvements, added --profile [`7486823`](https://github.com/RhetTbull/osxphotos/commit/74868238f3b1ee18feb744f137f5c14ef8e36ffc)
+
 #### [v0.44.10](https://github.com/RhetTbull/osxphotos/compare/v0.44.9...v0.44.10)
 
 > 22 January 2022

MANIFEST.in

@@ -1,7 +1,7 @@
-include README.md
+include osxphotos/*.json
-include README.rst
+include osxphotos/*.md
-include osxphotos/templates/*
 include osxphotos/phototemplate.tx
-include osxphotos/phototemplate.md
+include osxphotos/queries/*
-include osxphotos/tutorial.md
+include osxphotos/templates/*
-include osxphotos/queries/*
+include README.md
+include README.rst

README.md

@@ -38,6 +38,7 @@ OSXPhotos provides the ability to interact with and query Apple's Photos.app lib
   + [Raw Photos](#raw-photos)
   + [Template System](#template-system)
   + [ExifTool](#exiftoolExifTool)
+  + [PhotoExporter](#photoexporter)
   + [Text Detection](#textdetection)
   + [Utility Functions](#utility-functions)
 * [Examples](#examples)
@@ -600,6 +601,7 @@ Options:
                                   library, 2. system library, 3.
                                   ~/Pictures/Photos Library.photoslibrary
   -V, --verbose                   Print verbose output.
+  --timestamp                     Add time stamp to verbose output
   --keyword KEYWORD               Search for photos with keyword KEYWORD. If
                                   more than one keyword, treated as "OR", e.g.
                                   find photos matching any keyword
@@ -781,8 +783,15 @@ Options:
                                   folder.
   --deleted-only                  Include only photos from the 'Recently
                                   Deleted' folder.
-  --update                        Only export new or updated files. See notes
+  --update                        Only export new or updated files. See also
-                                  below on export and --update.
+                                  --force-update and notes below on export and
+                                  --update.
+  --force-update                  Only export new or updated files. Unlike
+                                  --update, --force-update will re-export photos
+                                  if their metadata has changed even if this
+                                  would not otherwise trigger an export. See
+                                  also --update and notes below on export and
+                                  --update.
   --ignore-signature              When used with '--update', ignores file
                                   signature when updating files. This is useful
                                   if you have processed or edited exported
@@ -1723,7 +1732,7 @@ Substitution                    Description
 {lf}                            A line feed: '\n', alias for {newline}
 {cr}                            A carriage return: '\r'
 {crlf}                          a carriage return + line feed: '\r\n'
-{osxphotos_version}             The osxphotos version, e.g. '0.44.12'
+{osxphotos_version}             The osxphotos version, e.g. '0.45.12'
 {osxphotos_cmd_line}            The full command line used to run osxphotos
 
 The following substitutions may result in multiple values. Thus if specified for
@@ -3627,7 +3636,7 @@ The following template field substitutions are availabe for use the templating s
 |{lf}|A line feed: '\n', alias for {newline}|
 |{cr}|A carriage return: '\r'|
 |{crlf}|a carriage return + line feed: '\r\n'|
-|{osxphotos_version}|The osxphotos version, e.g. '0.44.12'|
+|{osxphotos_version}|The osxphotos version, e.g. '0.45.12'|
 |{osxphotos_cmd_line}|The full command line used to run osxphotos|
 |{album}|Album(s) photo is contained in|
 |{folder_album}|Folder path + album photo is contained in. e.g. 'Folder/Subfolder/Album' or just 'Album' if no enclosing folder|
@@ -3711,6 +3720,105 @@ osxphotos.exiftool also provides an `ExifToolCaching` class which caches all met
 
 `ExifTool()` runs `exiftool` as a subprocess using the `-stay_open True` flag to keep the process running in the background.  The subprocess will be cleaned up when your main script terminates.  `ExifTool()` uses a singleton pattern to ensure that only one instance of `exiftool` is created.  Multiple instances of `ExifTool()` will all use the same `exiftool` subprocess.
 
+### <a name="photoexporter">PhotoExporter</a>
+
+[PhotoInfo.export()](#photoinfo) provides a simple method to export a photo.  This method actually calls `PhotoExporter.export()` to do the export.  `PhotoExporter` provides many more options to configure the export and report results and this is what the osxphotos command line export tools uses.
+
+#### `export(dest, filename=None, options: Optional[ExportOptions]=None) -> ExportResults`
+
+Export a photo.
+
+Args:
+
+- dest: must be valid destination path or exception raised
+- filename: (optional): name of exported picture; if not provided, will use current filename
+- options (ExportOptions): optional ExportOptions instance
+
+Returns: ExportResults instance
+
+*Note*: to use dry run mode, you must set options.dry_run=True and also pass in memory version of export_db, and no-op fileutil (e.g. ExportDBInMemory and FileUtilNoOp) in options.export_db and options.fileutil respectively.
+
+#### `ExportOptions`
+
+Options class for exporting photos with `export`
+
+Attributes:
+
+- convert_to_jpeg (bool): if True, converts non-jpeg images to jpeg
+- description_template (str): optional template string that will be rendered for use as photo description
+- download_missing: (bool, default=False): if True will attempt to export photo via applescript interaction with Photos if missing (see also use_photokit, use_photos_export)
+- dry_run: (bool, default=False): set to True to run in "dry run" mode
+- edited: (bool, default=False): if True will export the edited version of the photo otherwise exports the original version
+- exiftool_flags (list of str): optional list of flags to pass to exiftool when using exiftool option, e.g ["-m", "-F"]
+- exiftool: (bool, default = False): if True, will use exiftool to write metadata to export file
+- export_as_hardlink: (bool, default=False): if True, will hardlink files instead of copying them
+- export_db: (ExportDB_ABC): instance of a class that conforms to ExportDB_ABC with methods for getting/setting data related to exported files to compare update state
+- fileutil: (FileUtilABC): class that conforms to FileUtilABC with various file utilities
+- ignore_date_modified (bool): for use with sidecar and exiftool; if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
+- ignore_signature (bool, default=False): ignore file signature when used with update (look only at filename)
+- increment (bool, default=True): if True, will increment file name until a non-existant name is found if overwrite=False and increment=False, export will fail if destination file already exists
+- jpeg_ext (str): if set, will use this value for extension on jpegs converted to jpeg with convert_to_jpeg; if not set, uses jpeg; do not include the leading "."
+- jpeg_quality (float in range 0.0 <= jpeg_quality <= 1.0): a value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
+- keyword_template (list of str): list of template strings that will be rendered as used as keywords
+- live_photo (bool, default=False): if True, will also export the associated .mov for live photos
+- location (bool): if True, include location in exported metadata
+- merge_exif_keywords (bool): if True, merged keywords found in file's exif data (requires exiftool)
+- merge_exif_persons (bool): if True, merged persons found in file's exif data (requires exiftool)
+- overwrite (bool, default=False): if True will overwrite files if they already exist
+- persons (bool): if True, include persons in exported metadata
+- preview_suffix (str): optional string to append to end of filename for preview images
+- preview (bool): if True, also exports preview image
+- raw_photo (bool, default=False): if True, will also export the associated RAW photo
+- render_options (RenderOptions): optional osxphotos.phototemplate.RenderOptions instance to specify options for rendering templates
+- replace_keywords (bool): if True, keyword_template replaces any keywords, otherwise it's additive
+- sidecar_drop_ext (bool, default=False): if True, drops the photo's extension from sidecar filename (e.g. 'IMG_1234.json' instead of 'IMG_1234.JPG.json')
+- sidecar: bit field (int): set to one or more of SIDECAR_XMP, SIDECAR_JSON, SIDECAR_EXIFTOOL
+  - SIDECAR_JSON: if set will write a json sidecar with data in format readable by exiftool sidecar filename will be dest/filename.json; includes exiftool tag group names (e.g. `exiftool -G -j`)
+  - SIDECAR_EXIFTOOL: if set will write a json sidecar with data in format readable by exiftool sidecar filename will be dest/filename.json; does not include exiftool tag group names (e.g. `exiftool -j`)
+  - SIDECAR_XMP: if set will write an XMP sidecar with IPTC data sidecar filename will be dest/filename.xmp
+- strip (bool): if True, strip whitespace from rendered templates
+- timeout (int, default=120): timeout in seconds used with use_photos_export
+- touch_file (bool, default=False): if True, sets file's modification time upon photo date
+- update (bool, default=False): if True export will run in update mode, that is, it will not export the photo if the current version already exists in the destination
+- use_albums_as_keywords (bool, default = False): if True, will include album names in keywords when exporting metadata with exiftool or sidecar
+- use_persons_as_keywords (bool, default = False): if True, will include person names in keywords when exporting metadata with exiftool or sidecar
+- use_photos_export (bool, default=False): if True will attempt to export photo via applescript interaction with Photos even if not missing (see also use_photokit, download_missing)
+- use_photokit (bool, default=False): if True, will use photokit to export photos when use_photos_export is True
+- verbose (Callable): optional callable function to use for printing verbose text during processing; if None (default), does not print output.
+
+#### `ExportResults`
+
+`PhotoExporter().export()` returns an instance of this class.
+
+`ExportResults` has the following properties:
+
+- exported: list of all exported files (A single call to export could export more than one file, e.g. original file, preview, live video, raw, etc.)
+- new: list of new files exported when used with update=True
+- updated: list of updated files when used with update=True
+- skipped: list of skipped files when used with update=True
+- exif_updated: list of updated files when used with update=True and exiftool
+- touched: list of files touched during export (e.g. file date/time updated with touch_file=True)
+- to_touch: Reserved for internal use of export
+- converted_to_jpeg: list of files converted to jpeg when convert_to_jpeg=True
+- sidecar_json_written: list of JSON sidecars written
+- sidecar_json_skipped: list of JSON sidecars skipped when update=True
+- sidecar_exiftool_written: list of exiftool sidecars written
+- sidecar_exiftool_skipped: list of exiftool sidecars skipped when update=True
+- sidecar_xmp_written: list of XMP sidecars written
+- sidecar_xmp_skipped: list of XMP sidecars skipped when update=True
+- missing: list of missing files
+- error: list of tuples containing (filename, error) if error generated during export
+- exiftool_warning: list of warnings generated by exiftool during export
+- exiftool_error: list of errors generated by exiftool during export
+- xattr_written: list of files with extended attributes written during export
+- xattr_skipped: list of files where extended attributes were skipped when update=True
+- deleted_files: reserved for use by osxphotos CLI
+- deleted_directories: reserved for use by osxphotos CLI
+- exported_album: reserved for use by osxphotos CLI
+- skipped_album: reserved for use by osxphotos CLI
+- missing_album: reserved for use by osxphotos CLI 
+
+
 ### <a name="textdetection">Text Detection</a>
 
 The [PhotoInfo.detected_text()](#detected_text_method) and the `{detected_text}` template will perform text detection on the photos in your library. Text detection is a slow process so to avoid unnecessary re-processing of photos, osxphotos will cache the results of the text detection process as an extended attribute on the photo image file.  Extended attributes do not modify the actual file.  The extended attribute is named `osxphotos.metadata:detected_text` and can be viewed using the built-in [xattr](https://ss64.com/osx/xattr.html) command or my [osxmetadata](https://github.com/RhetTbull/osxmetadata) tool.  If you want to remove the cached attribute, you can do so with osxmetadata as follows:
@@ -3847,7 +3955,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
     <td align="center"><a href="https://github.com/mkirkland4874"><img src="https://avatars.githubusercontent.com/u/36466711?v=4?s=75" width="75px;" alt=""/><br /><sub><b>mkirkland4874</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Amkirkland4874" title="Bug reports">🐛</a> <a href="#example-mkirkland4874" title="Examples">💡</a></td>
     <td align="center"><a href="https://github.com/jcommisso07"><img src="https://avatars.githubusercontent.com/u/3111054?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Joseph Commisso</b></sub></a><br /><a href="#data-jcommisso07" title="Data">🔣</a></td>
     <td align="center"><a href="https://github.com/dssinger"><img src="https://avatars.githubusercontent.com/u/1817903?v=4?s=75" width="75px;" alt=""/><br /><sub><b>David Singer</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Adssinger" title="Bug reports">🐛</a></td>
-    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a></td>
+    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a> <a href="#ideas-oPromessa" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/RhetTbull/osxphotos/commits?author=oPromessa" title="Tests">⚠️</a></td>
     <td align="center"><a href="http://spencerchang.me"><img src="https://avatars.githubusercontent.com/u/14796580?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Spencer Chang</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Aspencerc99" title="Bug reports">🐛</a></td>
   </tr>
   <tr>
@@ -3873,7 +3981,6 @@ My goal is make osxphotos as reliable and comprehensive as possible.  The test s
 
 - Audio-only files are not handled.  It is possible to store audio-only files in Photos.  osxphotos currently only handles images and videos. See [Issue #436](https://github.com/RhetTbull/osxphotos/issues/436)
 - Face coordinates (mouth, left eye, right eye) may not be correct for images where the head is tilted.  See [Issue #196](https://github.com/RhetTbull/osxphotos/issues/196).
-- Raw images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the raw image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the raw image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75).
 
 ## Implementation Notes

docs/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 2b46ed6fff6c9124ce0554461f401ffd
+config: 30f35e310e3c2ebf8f561c37c515c685
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/_static/documentation_options.js

@@ -1,6 +1,6 @@
 var DOCUMENTATION_OPTIONS = {
     URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
-    VERSION: '0.44.12',
+    VERSION: '0.45.12',
     LANGUAGE: 'None',
     COLLAPSE_INDEX: false,
     BUILDER: 'html',

docs/cli.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.44.12 documentation</title>
+    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/genindex.html

@@ -5,7 +5,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Index &#8212; osxphotos 0.44.12 documentation</title>
+    <title>Index &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/index.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.44.12 documentation</title>
+    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/modules.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos &#8212; osxphotos 0.44.12 documentation</title>
+    <title>osxphotos &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/reference.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos package &#8212; osxphotos 0.44.12 documentation</title>
+    <title>osxphotos package &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/search.html

@@ -5,7 +5,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Search &#8212; osxphotos 0.44.12 documentation</title>
+    <title>Search &#8212; osxphotos 0.45.12 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     

osxphotos.spec

@@ -14,6 +14,7 @@ datas = [
     ("osxphotos/phototemplate.tx", "osxphotos"),
     ("osxphotos/phototemplate.md", "osxphotos"),
     ("osxphotos/tutorial.md", "osxphotos"),
+    ("osxphotos/exiftool_filetypes.json", "osxphotos"),
 ]
 package_imports = [["photoscript", ["photoscript.applescript"]]]
 for package, files in package_imports:

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.44.12"
+__version__ = "0.45.12"

osxphotos/cli.py

@@ -61,6 +61,7 @@ from .configoptions import (
     ConfigOptionsInvalidError,
     ConfigOptionsLoadError,
 )
+from .crash_reporter import crash_reporter
 from .datetime_formatter import DateTimeFormatter
 from .exiftool import get_exiftool_path
 from .export_db import ExportDB, ExportDBInMemory
@@ -79,7 +80,7 @@ from .sqlgrep import sqlgrep
 from .uti import get_preferred_uti_extension
 from .utils import (
     expand_and_validate_filepath,
-    list_directory,
+    format_sec_to_hhmmss,
     load_function,
     normalize_fs_path,
 )
@@ -133,6 +134,11 @@ __all__ = [
 # global variable to control verbose output
 # set via --verbose/-V
 VERBOSE = False
+VERBOSE_TIMESTAMP = False
+
+# global variable to control debug output
+# set via --debug
+DEBUG = False
 
 # used to show/hide hidden commands
 OSXPHOTOS_HIDDEN = not bool(os.getenv("OSXPHOTOS_SHOW_HIDDEN", default=False))
@@ -140,6 +146,9 @@ OSXPHOTOS_HIDDEN = not bool(os.getenv("OSXPHOTOS_SHOW_HIDDEN", default=False))
 # used by snap and diff commands
 OSXPHOTOS_SNAPSHOT_DIR = "/private/tmp/osxphotos_snapshots"
 
+# where to write the crash report if osxphotos crashes
+OSXPHOTOS_CRASH_LOG = os.getcwd() + "/osxphotos_crash.log"
+
 rich.traceback.install()
 
 
@@ -147,8 +156,10 @@ def verbose_(*args, **kwargs):
     """print output if verbose flag set"""
     if VERBOSE:
         styled_args = []
+        timestamp = str(datetime.datetime.now()) + " -- " if VERBOSE_TIMESTAMP else ""
         for arg in args:
             if type(arg) == str:
+                arg = timestamp + arg
                 if "error" in arg.lower():
                     arg = click.style(arg, fg=CLI_COLOR_ERROR)
                 elif "warning" in arg.lower():
@@ -671,12 +682,13 @@ def QUERY_OPTIONS(f):
 @click.version_option(__version__, "--version", "-v")
 @click.pass_context
 def cli(ctx, db, json_, debug):
-    ctx.obj = CLI_Obj(db=db, json=json_, debug=debug)
+    ctx.obj = CLI_Obj(db=db, json=json_)
 
 
 @cli.command(cls=ExportCommand)
 @DB_OPTION
 @click.option("--verbose", "-V", "verbose", is_flag=True, help="Print verbose output.")
+@click.option("--timestamp", is_flag=True, help="Add time stamp to verbose output")
 @QUERY_OPTIONS
 @click.option(
     "--missing",
@@ -687,7 +699,15 @@ def cli(ctx, db, json_, debug):
 @click.option(
     "--update",
     is_flag=True,
-    help="Only export new or updated files. See notes below on export and --update.",
+    help="Only export new or updated files. "
+    "See also --force-update and notes below on export and --update.",
+)
+@click.option(
+    "--force-update",
+    is_flag=True,
+    help="Only export new or updated files. Unlike --update, --force-update will re-export photos "
+    "if their metadata has changed even if this would not otherwise trigger an export. "
+    "See also --update and notes below on export and --update.",
 )
 @click.option(
     "--ignore-signature",
@@ -1193,10 +1213,25 @@ def cli(ctx, db, json_, debug):
     f"Can be specified multiple times. Valid options are: {PROFILE_SORT_KEYS}. "
     "Default = 'cumulative'.",
 )
+@click.option(
+    "--debug",
+    required=False,
+    is_flag=True,
+    default=False,
+    hidden=OSXPHOTOS_HIDDEN,
+    help="Enable debug output.",
+)
 @DB_ARGUMENT
 @click.argument("dest", nargs=1, type=click.Path(exists=True))
 @click.pass_obj
 @click.pass_context
+@crash_reporter(
+    OSXPHOTOS_CRASH_LOG,
+    "[red]Something went wrong and osxphotos encountered an error:[/red]",
+    "osxphotos crash log",
+    "Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.",
+    f"osxphotos version: {__version__}",
+)
 def export(
     ctx,
     cli_obj,
@@ -1228,8 +1263,10 @@ def export(
     from_time,
     to_time,
     verbose,
+    timestamp,
     missing,
     update,
+    force_update,
     ignore_signature,
     only_new,
     dry_run,
@@ -1333,6 +1370,7 @@ def export(
     preview_if_missing,
     profile,
     profile_sort,
+    debug,
 ):
     """Export photos from the Photos database.
     Export path DEST is required.
@@ -1346,6 +1384,11 @@ def export(
     to modify this behavior.
     """
 
+    global DEBUG
+    if debug:
+        DEBUG = True
+        osxphotos._set_debug(True)
+
     if profile:
         click.echo("Profiling...")
         profile_sort = profile_sort or ["cumulative"]
@@ -1374,7 +1417,9 @@ def export(
     )
 
     global VERBOSE
+    global VERBOSE_TIMESTAMP
     VERBOSE = bool(verbose)
+    VERBOSE_TIMESTAMP = timestamp
 
     if load_config:
         try:
@@ -1387,137 +1432,138 @@ def export(
                 ),
                 err=True,
             )
-            raise click.Abort()
+            sys.exit(1)
 
         # re-set the local vars to the corresponding config value
         # this isn't elegant but avoids having to rewrite this function to use cfg.varname for every parameter
-        db = cfg.db
+        add_exported_to_album = cfg.add_exported_to_album
-        photos_library = cfg.photos_library
+        add_missing_to_album = cfg.add_missing_to_album
-        keyword = cfg.keyword
+        add_skipped_to_album = cfg.add_skipped_to_album
-        person = cfg.person
         album = cfg.album
-        folder = cfg.folder
-        name = cfg.name
-        uuid = cfg.uuid
-        uuid_from_file = cfg.uuid_from_file
-        title = cfg.title
-        no_title = cfg.no_title
-        description = cfg.description
-        no_description = cfg.no_description
-        uti = cfg.uti
-        ignore_case = cfg.ignore_case
-        edited = cfg.edited
-        external_edit = cfg.external_edit
-        favorite = cfg.favorite
-        not_favorite = cfg.not_favorite
-        hidden = cfg.hidden
-        not_hidden = cfg.not_hidden
-        shared = cfg.shared
-        not_shared = cfg.not_shared
-        from_date = cfg.from_date
-        to_date = cfg.to_date
-        from_time = cfg.from_time
-        to_time = cfg.to_time
-        verbose = cfg.verbose
-        missing = cfg.missing
-        update = cfg.update
-        ignore_signature = cfg.ignore_signature
-        dry_run = cfg.dry_run
-        export_as_hardlink = cfg.export_as_hardlink
-        touch_file = cfg.touch_file
-        overwrite = cfg.overwrite
-        retry = cfg.retry
-        export_by_date = cfg.export_by_date
-        skip_edited = cfg.skip_edited
-        skip_original_if_edited = cfg.skip_original_if_edited
-        skip_bursts = cfg.skip_bursts
-        skip_live = cfg.skip_live
-        skip_raw = cfg.skip_raw
-        skip_uuid = cfg.skip_uuid
-        skip_uuid_from_file = cfg.skip_uuid_from_file
-        person_keyword = cfg.person_keyword
         album_keyword = cfg.album_keyword
-        keyword_template = cfg.keyword_template
+        beta = cfg.beta
-        replace_keywords = cfg.replace_keywords
-        description_template = cfg.description_template
-        finder_tag_template = cfg.finder_tag_template
-        finder_tag_keywords = cfg.finder_tag_keywords
-        xattr_template = cfg.xattr_template
-        current_name = cfg.current_name
-        convert_to_jpeg = cfg.convert_to_jpeg
-        jpeg_quality = cfg.jpeg_quality
-        sidecar = cfg.sidecar
-        sidecar_drop_ext = cfg.sidecar_drop_ext
-        only_photos = cfg.only_photos
-        only_movies = cfg.only_movies
         burst = cfg.burst
-        not_burst = cfg.not_burst
+        cleanup = cfg.cleanup
-        live = cfg.live
+        convert_to_jpeg = cfg.convert_to_jpeg
-        not_live = cfg.not_live
+        current_name = cfg.current_name
-        download_missing = cfg.download_missing
+        db = cfg.db
-        exiftool = cfg.exiftool
-        exiftool_path = cfg.exiftool_path
-        exiftool_option = cfg.exiftool_option
-        exiftool_merge_keywords = cfg.exiftool_merge_keywords
-        exiftool_merge_persons = cfg.exiftool_merge_persons
-        ignore_date_modified = cfg.ignore_date_modified
-        portrait = cfg.portrait
-        not_portrait = cfg.not_portrait
-        screenshot = cfg.screenshot
-        not_screenshot = cfg.not_screenshot
-        slow_mo = cfg.slow_mo
-        not_slow_mo = cfg.not_slow_mo
-        time_lapse = cfg.time_lapse
-        not_time_lapse = cfg.not_time_lapse
-        hdr = cfg.hdr
-        not_hdr = cfg.not_hdr
-        selfie = cfg.selfie
-        not_selfie = cfg.not_selfie
-        panorama = cfg.panorama
-        not_panorama = cfg.not_panorama
-        has_raw = cfg.has_raw
-        directory = cfg.directory
-        filename_template = cfg.filename_template
-        jpeg_ext = cfg.jpeg_ext
-        strip = cfg.strip
-        edited_suffix = cfg.edited_suffix
-        original_suffix = cfg.original_suffix
-        place = cfg.place
-        no_place = cfg.no_place
-        location = cfg.location
-        no_location = cfg.no_location
-        has_comment = cfg.has_comment
-        no_comment = cfg.no_comment
-        has_likes = cfg.has_likes
-        no_likes = cfg.no_likes
-        label = cfg.label
         deleted = cfg.deleted
         deleted_only = cfg.deleted_only
-        use_photos_export = cfg.use_photos_export
+        description = cfg.description
-        use_photokit = cfg.use_photokit
+        description_template = cfg.description_template
-        report = cfg.report
+        directory = cfg.directory
-        cleanup = cfg.cleanup
+        download_missing = cfg.download_missing
-        add_exported_to_album = cfg.add_exported_to_album
+        dry_run = cfg.dry_run
-        add_skipped_to_album = cfg.add_skipped_to_album
-        add_missing_to_album = cfg.add_missing_to_album
-        exportdb = cfg.exportdb
-        beta = cfg.beta
-        only_new = cfg.only_new
-        in_album = cfg.in_album
-        not_in_album = cfg.not_in_album
-        min_size = cfg.min_size
-        max_size = cfg.max_size
-        regex = cfg.regex
-        selected = cfg.selected
-        exif = cfg.exif
-        query_eval = cfg.query_eval
-        query_function = cfg.query_function
         duplicate = cfg.duplicate
+        edited = cfg.edited
+        edited_suffix = cfg.edited_suffix
+        exif = cfg.exif
+        exiftool = cfg.exiftool
+        exiftool_merge_keywords = cfg.exiftool_merge_keywords
+        exiftool_merge_persons = cfg.exiftool_merge_persons
+        exiftool_option = cfg.exiftool_option
+        exiftool_path = cfg.exiftool_path
+        export_as_hardlink = cfg.export_as_hardlink
+        export_by_date = cfg.export_by_date
+        exportdb = cfg.exportdb
+        external_edit = cfg.external_edit
+        favorite = cfg.favorite
+        filename_template = cfg.filename_template
+        finder_tag_keywords = cfg.finder_tag_keywords
+        finder_tag_template = cfg.finder_tag_template
+        folder = cfg.folder
+        force_update = cfg.force_update
+        from_date = cfg.from_date
+        from_time = cfg.from_time
+        has_comment = cfg.has_comment
+        has_likes = cfg.has_likes
+        has_raw = cfg.has_raw
+        hdr = cfg.hdr
+        hidden = cfg.hidden
+        ignore_case = cfg.ignore_case
+        ignore_date_modified = cfg.ignore_date_modified
+        ignore_signature = cfg.ignore_signature
+        in_album = cfg.in_album
+        jpeg_ext = cfg.jpeg_ext
+        jpeg_quality = cfg.jpeg_quality
+        keyword = cfg.keyword
+        keyword_template = cfg.keyword_template
+        label = cfg.label
+        live = cfg.live
+        location = cfg.location
+        max_size = cfg.max_size
+        min_size = cfg.min_size
+        missing = cfg.missing
+        name = cfg.name
+        no_comment = cfg.no_comment
+        no_description = cfg.no_description
+        no_likes = cfg.no_likes
+        no_location = cfg.no_location
+        no_place = cfg.no_place
+        no_title = cfg.no_title
+        not_burst = cfg.not_burst
+        not_favorite = cfg.not_favorite
+        not_hdr = cfg.not_hdr
+        not_hidden = cfg.not_hidden
+        not_in_album = cfg.not_in_album
+        not_live = cfg.not_live
+        not_panorama = cfg.not_panorama
+        not_portrait = cfg.not_portrait
+        not_screenshot = cfg.not_screenshot
+        not_selfie = cfg.not_selfie
+        not_shared = cfg.not_shared
+        not_slow_mo = cfg.not_slow_mo
+        not_time_lapse = cfg.not_time_lapse
+        only_movies = cfg.only_movies
+        only_new = cfg.only_new
+        only_photos = cfg.only_photos
+        original_suffix = cfg.original_suffix
+        overwrite = cfg.overwrite
+        panorama = cfg.panorama
+        person = cfg.person
+        person_keyword = cfg.person_keyword
+        photos_library = cfg.photos_library
+        place = cfg.place
+        portrait = cfg.portrait
         post_command = cfg.post_command
         post_function = cfg.post_function
         preview = cfg.preview
-        preview_suffix = cfg.preview_suffix
         preview_if_missing = cfg.preview_if_missing
+        preview_suffix = cfg.preview_suffix
+        query_eval = cfg.query_eval
+        query_function = cfg.query_function
+        regex = cfg.regex
+        replace_keywords = cfg.replace_keywords
+        report = cfg.report
+        retry = cfg.retry
+        screenshot = cfg.screenshot
+        selected = cfg.selected
+        selfie = cfg.selfie
+        shared = cfg.shared
+        sidecar = cfg.sidecar
+        sidecar_drop_ext = cfg.sidecar_drop_ext
+        skip_bursts = cfg.skip_bursts
+        skip_edited = cfg.skip_edited
+        skip_live = cfg.skip_live
+        skip_original_if_edited = cfg.skip_original_if_edited
+        skip_raw = cfg.skip_raw
+        skip_uuid = cfg.skip_uuid
+        skip_uuid_from_file = cfg.skip_uuid_from_file
+        slow_mo = cfg.slow_mo
+        strip = cfg.strip
+        time_lapse = cfg.time_lapse
+        title = cfg.title
+        to_date = cfg.to_date
+        to_time = cfg.to_time
+        touch_file = cfg.touch_file
+        update = cfg.update
+        use_photokit = cfg.use_photokit
+        use_photos_export = cfg.use_photos_export
+        uti = cfg.uti
+        uuid = cfg.uuid
+        uuid_from_file = cfg.uuid_from_file
+        verbose = cfg.verbose
+        xattr_template = cfg.xattr_template
 
         # config file might have changed verbose
         VERBOSE = bool(verbose)
@@ -1557,8 +1603,8 @@ def export(
     dependent_options = [
         ("missing", ("download_missing", "use_photos_export")),
         ("jpeg_quality", ("convert_to_jpeg")),
-        ("ignore_signature", ("update")),
+        ("ignore_signature", ("update", "force_update")),
-        ("only_new", ("update")),
+        ("only_new", ("update", "force_update")),
         ("exiftool_option", ("exiftool")),
         ("exiftool_merge_keywords", ("exiftool", "sidecar")),
         ("exiftool_merge_persons", ("exiftool", "sidecar")),
@@ -1572,7 +1618,7 @@ def export(
             ),
             err=True,
         )
-        raise click.Abort()
+        sys.exit(1)
 
     if all(x in [s.lower() for s in sidecar] for x in ["json", "exiftool"]):
         click.echo(
@@ -1582,7 +1628,7 @@ def export(
             ),
             err=True,
         )
-        raise click.Abort()
+        sys.exit(1)
 
     if xattr_template:
         for attr, _ in xattr_template:
@@ -1595,7 +1641,7 @@ def export(
                     ),
                     err=True,
                 )
-                raise click.Abort()
+                sys.exit(1)
 
     if save_config:
         verbose_(f"Saving options to file {save_config}")
@@ -1616,7 +1662,7 @@ def export(
         click.echo(
             click.style(f"DEST {dest} must be valid path", fg=CLI_COLOR_ERROR), err=True
         )
-        raise click.Abort()
+        sys.exit(1)
 
     dest = str(pathlib.Path(dest).resolve())
 
@@ -1627,7 +1673,7 @@ def export(
             ),
             err=True,
         )
-        raise click.Abort()
+        sys.exit(1)
 
     # if use_photokit and not check_photokit_authorization():
     #     click.echo(
@@ -1889,56 +1935,61 @@ def export(
             else None
         )
 
+        photo_num = 0
         # send progress bar output to /dev/null if verbose to hide the progress bar
         fp = open(os.devnull, "w") if verbose else None
-        with click.progressbar(photos, file=fp) as bar:
+        with click.progressbar(photos, show_pos=True, file=fp) as bar:
             for p in bar:
+                photo_num += 1
                 export_results = export_photo(
                     photo=p,
                     dest=dest,
-                    verbose=verbose,
+                    album_keyword=album_keyword,
-                    export_by_date=export_by_date,
+                    convert_to_jpeg=convert_to_jpeg,
-                    sidecar=sidecar,
+                    description_template=description_template,
-                    sidecar_drop_ext=sidecar_drop_ext,
+                    directory=directory,
-                    update=update,
-                    ignore_signature=ignore_signature,
-                    export_as_hardlink=export_as_hardlink,
-                    overwrite=overwrite,
-                    export_edited=export_edited,
-                    skip_original_if_edited=skip_original_if_edited,
-                    original_name=original_name,
-                    export_live=export_live,
                     download_missing=download_missing,
-                    exiftool=exiftool,
+                    dry_run=dry_run,
+                    edited_suffix=edited_suffix,
                     exiftool_merge_keywords=exiftool_merge_keywords,
                     exiftool_merge_persons=exiftool_merge_persons,
-                    directory=directory,
-                    filename_template=filename_template,
-                    export_raw=export_raw,
-                    album_keyword=album_keyword,
-                    person_keyword=person_keyword,
-                    keyword_template=keyword_template,
-                    description_template=description_template,
-                    export_db=export_db,
-                    fileutil=fileutil,
-                    dry_run=dry_run,
-                    touch_file=touch_file,
-                    edited_suffix=edited_suffix,
-                    original_suffix=original_suffix,
-                    use_photos_export=use_photos_export,
-                    convert_to_jpeg=convert_to_jpeg,
-                    jpeg_quality=jpeg_quality,
-                    ignore_date_modified=ignore_date_modified,
-                    use_photokit=use_photokit,
                     exiftool_option=exiftool_option,
-                    strip=strip,
+                    exiftool=exiftool,
+                    export_as_hardlink=export_as_hardlink,
+                    export_by_date=export_by_date,
+                    export_db=export_db,
+                    export_dir=dest,
+                    export_edited=export_edited,
+                    export_live=export_live,
+                    export_preview=preview,
+                    export_raw=export_raw,
+                    filename_template=filename_template,
+                    fileutil=fileutil,
+                    force_update=force_update,
+                    ignore_date_modified=ignore_date_modified,
+                    ignore_signature=ignore_signature,
                     jpeg_ext=jpeg_ext,
+                    jpeg_quality=jpeg_quality,
+                    keyword_template=keyword_template,
+                    num_photos=num_photos,
+                    original_name=original_name,
+                    original_suffix=original_suffix,
+                    overwrite=overwrite,
+                    person_keyword=person_keyword,
+                    photo_num=photo_num,
+                    preview_if_missing=preview_if_missing,
+                    preview_suffix=preview_suffix,
                     replace_keywords=replace_keywords,
                     retry=retry,
-                    export_dir=dest,
+                    sidecar_drop_ext=sidecar_drop_ext,
-                    export_preview=preview,
+                    sidecar=sidecar,
-                    preview_suffix=preview_suffix,
+                    skip_original_if_edited=skip_original_if_edited,
-                    preview_if_missing=preview_if_missing,
+                    strip=strip,
+                    touch_file=touch_file,
+                    update=update,
+                    use_photokit=use_photokit,
+                    use_photos_export=use_photos_export,
+                    verbose=verbose,
                 )
 
                 if post_function:
@@ -2032,7 +2083,6 @@ def export(
                         finder_tag_template=finder_tag_template,
                         strip=strip,
                         export_dir=dest,
-                        export_db=export_db,
                     )
                     results.xattr_written.extend(tags_written)
                     results.xattr_skipped.extend(tags_skipped)
@@ -2044,7 +2094,6 @@ def export(
                         xattr_template,
                         strip=strip,
                         export_dir=dest,
-                        export_db=export_db,
                     )
                     results.xattr_written.extend(xattr_written)
                     results.xattr_skipped.extend(xattr_skipped)
@@ -2053,7 +2102,7 @@ def export(
             fp.close()
 
         photo_str_total = "photos" if len(photos) != 1 else "photo"
-        if update:
+        if update or force_update:
             summary = (
                 f"Processed: {len(photos)} {photo_str_total}, "
                 f"exported: {len(results.new)}, "
@@ -2072,12 +2121,14 @@ def export(
             summary += f", touched date: {len(results.touched)}"
         click.echo(summary)
         stop_time = time.perf_counter()
-        click.echo(f"Elapsed time: {(stop_time-start_time):.3f} seconds")
+        click.echo(f"Elapsed time: {format_sec_to_hhmmss(stop_time-start_time)}")
     else:
         click.echo("Did not find any photos to export")
 
     # cleanup files and do report if needed
     if cleanup:
+        db_file = str(pathlib.Path(export_db_path).resolve())
+        db_files = [db_file, db_file + "-wal", db_file + "-shm"]
         all_files = (
             results.exported
             + results.skipped
@@ -2096,7 +2147,7 @@ def export(
             + results.missing
             # include files that have error in case they exist from previous export
             + [r[0] for r in results.error]
-            + [str(pathlib.Path(export_db_path).resolve())]
+            + db_files
         )
         click.echo(f"Cleaning up {dest}")
         cleaned_files, cleaned_dirs = cleanup_files(dest, all_files, fileutil)
@@ -2581,6 +2632,7 @@ def export_photo(
     sidecar=None,
     sidecar_drop_ext=False,
     update=None,
+    force_update=None,
     ignore_signature=None,
     export_as_hardlink=None,
     overwrite=None,
@@ -2619,51 +2671,55 @@ def export_photo(
     export_preview=False,
     preview_suffix=None,
     preview_if_missing=False,
+    photo_num=1,
+    num_photos=1,
 ):
     """Helper function for export that does the actual export
 
     Args:
         photo: PhotoInfo object
         dest: destination path as string
-        verbose: boolean; print verbose output
+        album_keyword: bool; if True, exports album names as keywords in metadata
-        export_by_date: boolean; create export folder in form dest/YYYY/MM/DD
+        convert_to_jpeg: bool; if True, converts non-jpeg images to jpeg
-        sidecar: list zero, 1 or 2 of ["json","xmp"] of sidecar variety to export
+        description_template: str; optional template string that will be rendered for use as photo description
-        sidecar_drop_ext: boolean; if True, drops photo extension from sidecar name
-        export_as_hardlink: boolean; hardlink files instead of copying them
-        overwrite: boolean; overwrite dest file if it already exists
-        original_name: boolean; use original filename instead of current filename
-        export_live: boolean; also export live video component if photo is a live photo
-                     live video will have same name as photo but with .mov extension
-        download_missing: attempt download of missing iCloud photos
-        exiftool: use exiftool to write EXIF metadata directly to exported photo
         directory: template used to determine output directory
-        filename_template: template use to determine output file
+        download_missing: attempt download of missing iCloud photos
-        export_raw: boolean; if True exports raw image associate with the photo
+        dry_run: bool; if True, doesn't actually export or update any files
-        export_edited: boolean; if True exports edited version of photo if there is one
+        exiftool_merge_keywords: bool; if True, merged keywords found in file's exif data (requires exiftool)
-        skip_original_if_edited: boolean; if True does not export original if photo has been edited
+        exiftool_merge_persons: bool; if True, merged persons found in file's exif data (requires exiftool)
-        album_keyword: boolean; if True, exports album names as keywords in metadata
-        person_keyword: boolean; if True, exports person names as keywords in metadata
-        keyword_template: list of strings; if provided use rendered template strings as keywords
-        description_template: string; optional template string that will be rendered for use as photo description
-        export_db: export database instance compatible with ExportDB_ABC
-        fileutil: file util class compatible with FileUtilABC
-        dry_run: boolean; if True, doesn't actually export or update any files
-        touch_file: boolean; sets file's modification time to match photo date
-        use_photos_export: boolean; if True forces the use of AppleScript to export even if photo not missing
-        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
-        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
-        ignore_date_modified: if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
         exiftool_option: optional list flags (e.g. ["-m", "-F"]) to pass to exiftool
-        exiftool_merge_keywords: boolean; if True, merged keywords found in file's exif data (requires exiftool)
+        exiftool: bool; use exiftool to write EXIF metadata directly to exported photo
-        exiftool_merge_persons: boolean; if True, merged persons found in file's exif data (requires exiftool)
+        export_as_hardlink: bool; hardlink files instead of copying them
+        export_by_date: bool; create export folder in form dest/YYYY/MM/DD
+        export_db: export database instance compatible with ExportDB_ABC
+        export_dir: top-level export directory for {export_dir} template
+        export_edited: bool; if True exports edited version of photo if there is one
+        export_live: bool; also export live video component if photo is a live photo; live video will have same name as photo but with .mov extension
+        export_preview: export the preview image generated by Photos
+        export_raw: bool; if True exports raw image associate with the photo
+        filename_template: template use to determine output file
+        fileutil: file util class compatible with FileUtilABC
+        force_update: bool, only export updated photos but trigger export even if only metadata has changed
+        ignore_date_modified: if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
         jpeg_ext: if not None, specify the extension to use for all JPEG images on export
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
+        keyword_template: list of strings; if provided use rendered template strings as keywords
+        num_photos: int, total number of photos that will be exported
+        original_name: bool; use original filename instead of current filename
+        overwrite: bool; overwrite dest file if it already exists
+        person_keyword: bool; if True, exports person names as keywords in metadata
+        photo_num: int, which number photo in total of num_photos is being exported
+        preview_if_missing: bool, export preview if original is missing
+        preview_suffix: str, template to use as suffix for preview images
         replace_keywords: if True, --keyword-template replaces keywords instead of adding keywords
         retry: retry up to retry # of times if there's an error
-        export_dir: top-level export directory for {export_dir} template
+        sidecar_drop_ext: bool; if True, drops photo extension from sidecar name
-        export_preview: export the preview image generated by Photos
+        sidecar: list zero, 1 or 2 of ["json","xmp"] of sidecar variety to export
-        preview_suffix: str, template to use as suffix for preview images
+        skip_original_if_edited: bool; if True does not export original if photo has been edited
-        preview_if_missing: bool, export preview if original is missing
+        touch_file: bool; sets file's modification time to match photo date
-
+        update: bool, only export updated photos
+        use_photos_export: bool; if True forces the use of AppleScript to export even if photo not missing
+        verbose: bool; print verbose output
     Returns:
         list of path(s) of exported photo or None if photo was missing
 
@@ -2784,7 +2840,7 @@ def export_photo(
             original_filename = str(original_filename)
 
             verbose_(
-                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename}"
+                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename} ({photo_num}/{num_photos})"
             )
 
             results += export_photo_to_directory(
@@ -2809,6 +2865,7 @@ def export_photo(
                 export_raw=export_raw,
                 filename=original_filename,
                 fileutil=fileutil,
+                force_update=force_update,
                 ignore_date_modified=ignore_date_modified,
                 ignore_signature=ignore_signature,
                 jpeg_ext=jpeg_ext,
@@ -2921,6 +2978,7 @@ def export_photo(
                     export_raw=not export_original and export_raw,
                     filename=edited_filename,
                     fileutil=fileutil,
+                    force_update=force_update,
                     ignore_date_modified=ignore_date_modified,
                     ignore_signature=ignore_signature,
                     jpeg_ext=jpeg_ext,
@@ -2958,7 +3016,7 @@ def _render_suffix_template(
         return ""
 
     try:
-        options = RenderOptions(filename=True, export_dir=dest, exportdb=export_db)
+        options = RenderOptions(filename=True, export_dir=dest)
         rendered_suffix, unmatched = photo.render_template(suffix_template, options)
     except ValueError as e:
         raise click.BadOptionUsage(
@@ -3004,6 +3062,7 @@ def export_photo_to_directory(
     export_raw,
     filename,
     fileutil,
+    force_update,
     ignore_date_modified,
     ignore_signature,
     jpeg_ext,
@@ -3028,54 +3087,23 @@ def export_photo_to_directory(
     """Export photo to directory dest_path"""
 
     results = ExportResults()
-    # TODO: can be updated to let export2 do all the missing logic
+
-    if export_original:
+    # don't try to export photos in the trash if they're missing
-        if missing and not preview_if_missing:
+    photo_path = photo.path if export_original else photo.path_edited
-            space = " " if not verbose else ""
+    if photo.intrash and not photo_path and not preview_if_missing:
-            verbose_(
+        # skip deleted files if they're missing
-                f"{space}Skipping missing photo {photo.original_filename} ({photo.uuid})"
+        # as AppleScript/PhotoKit cannot export deleted photos
-            )
+        verbose_(
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
+            f"Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
-        elif (
+        )
-            photo.intrash
+        results.missing.append(str(pathlib.Path(dest_path) / filename))
-            and (not photo.path or (download_missing or use_photos_export))
+        return results
-            and not preview_if_missing
+
-        ):
+    render_options = RenderOptions(export_dir=export_dir, dest_path=dest_path)
-            # skip deleted files if they're missing or using use_photos_export
+
-            # as AppleScript/PhotoKit cannot export deleted photos
+    if not export_original and not edited:
-            space = " " if not verbose else ""
-            verbose_(
-                f"{space}Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
-            )
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results
-    elif not edited:
         verbose_(f"Skipping original version of {photo.original_filename}")
         return results
-    else:
-        # exporting the edited version
-        if missing and not preview_if_missing:
-            space = " " if not verbose else ""
-            verbose_(f"{space}Skipping missing edited photo for {filename}")
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results
-        elif (
-            photo.intrash
-            and (not photo.path_edited or (download_missing or use_photos_export))
-            and not preview_if_missing
-        ):
-            # skip deleted files if they're missing or using use_photos_export
-            # as AppleScript/PhotoKit cannot export deleted photos
-            space = " " if not verbose else ""
-            verbose_(
-                f"{space}Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
-            )
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results
-
-    render_options = RenderOptions(
-        export_dir=export_dir, dest_path=dest_path, exportdb=export_db
-    )
 
     tries = 0
     while tries <= retry:
@@ -3093,6 +3121,7 @@ def export_photo_to_directory(
                 export_as_hardlink=export_as_hardlink,
                 export_db=export_db,
                 fileutil=fileutil,
+                force_update=force_update,
                 ignore_date_modified=ignore_date_modified,
                 ignore_signature=ignore_signature,
                 jpeg_ext=jpeg_ext,
@@ -3118,7 +3147,7 @@ def export_photo_to_directory(
                 verbose=verbose_,
             )
             exporter = PhotoExporter(photo)
-            export_results = exporter.export2(
+            export_results = exporter.export(
                 dest=dest_path, filename=filename, options=export_options
             )
             for warning_ in export_results.exiftool_warning:
@@ -3148,6 +3177,9 @@ def export_photo_to_directory(
                     f"Retrying export for photo ({photo.uuid}: {photo.original_filename})"
                 )
         except Exception as e:
+            if DEBUG:
+                # if debug mode, don't swallow the exceptions
+                raise e
             click.echo(
                 click.style(
                     f"Error exporting photo ({photo.uuid}: {photo.original_filename}) as {filename}: {e}",
@@ -3164,7 +3196,7 @@ def export_photo_to_directory(
                 )
 
     if verbose:
-        if update:
+        if update or force_update:
             for new in results.new:
                 verbose_(f"Exported new file {new}")
             for updated in results.updated:
@@ -3195,7 +3227,7 @@ def get_filenames_from_template(
     Args:
         photo: a PhotoInfo instance
         filename_template: a PhotoTemplate template string, may be None
-        original_name: boolean; if True, use photo's original filename instead of current filename
+        original_name: bool; if True, use photo's original filename instead of current filename
         dest_path: the path the photo will be exported to
         strip: if True, strips leading/trailing white space from resulting template
         edited: if True, sets {edited_version} field to True, otherwise it gets set to False; set if you want template evaluated for edited version
@@ -3215,7 +3247,6 @@ def get_filenames_from_template(
                 edited_version=edited,
                 export_dir=export_dir,
                 dest_path=dest_path,
-                exportdb=export_db,
             )
             filenames, unmatched = photo.render_template(filename_template, options)
         except ValueError as e:
@@ -3257,9 +3288,9 @@ def get_dirnames_from_template(
     Args:
         photo: a PhotoInstance object
         directory: a PhotoTemplate template string, may be None
-        export_by_date: boolean; if True, creates output directories in form YYYY-MM-DD
+        export_by_date: bool; if True, creates output directories in form YYYY-MM-DD
         dest: top-level destination directory
-        dry_run: boolean; if True, runs in dry-run mode and does not create output directories
+        dry_run: bool; if True, runs in dry-run mode and does not create output directories
         strip: if True, strips leading/trailing white space from resulting template
         edited: if True, sets {edited_version} field to True, otherwise it gets set to False; set if you want template evaluated for edited version
 
@@ -3281,9 +3312,7 @@ def get_dirnames_from_template(
     elif directory:
         # got a directory template, render it and check results are valid
         try:
-            options = RenderOptions(
+            options = RenderOptions(dirname=True, edited_version=edited)
-                dirname=True, edited_version=edited, exportdb=export_db
-            )
             dirnames, unmatched = photo.render_template(directory, options)
         except ValueError as e:
             raise click.BadOptionUsage(
@@ -3519,7 +3548,7 @@ def write_export_report(report_file, results):
             click.style("Could not open output file for writing", fg=CLI_COLOR_ERROR),
             err=True,
         )
-        raise click.Abort()
+        sys.exit(1)
 
 
 def cleanup_files(dest_path, files_to_keep, fileutil):
@@ -3569,7 +3598,6 @@ def write_finder_tags(
     finder_tag_template=None,
     strip=False,
     export_dir=None,
-    export_db=None,
 ):
     """Write Finder tags (extended attributes) to files; only writes attributes if attributes on file differ from what would be written
 
@@ -3583,7 +3611,6 @@ def write_finder_tags(
         exiftool_merge_keywords: if True, include any keywords in the exif data of the source image as keywords
         finder_tag_template: list of templates to evaluate for determining Finder tags
         export_dir: value to use for {export_dir} template
-        export_db: an ExportDB object
 
     Returns:
         (list of file paths that were updated with new Finder tags, list of file paths skipped because Finder tags didn't need updating)
@@ -3615,7 +3642,6 @@ def write_finder_tags(
                     none_str=_OSXPHOTOS_NONE_SENTINEL,
                     path_sep="/",
                     export_dir=export_dir,
-                    exportdb=export_db,
                 )
                 rendered, unmatched = photo.render_template(template_str, options)
             except ValueError as e:
@@ -3663,7 +3689,6 @@ def write_extended_attributes(
     xattr_template,
     strip=False,
     export_dir=None,
-    export_db=None,
 ):
     """Writes extended attributes to exported files
 
@@ -3671,7 +3696,6 @@ def write_extended_attributes(
         photo: a PhotoInfo object
         strip:   xattr_template: list of tuples: (attribute name, attribute template)
         export_dir: value to use for {export_dir} template
-        exportdb: an ExportDB object
 
     Returns:
         tuple(list of file paths that were updated with new attributes, list of file paths skipped because attributes didn't need updating)
@@ -3681,10 +3705,7 @@ def write_extended_attributes(
     for xattr, template_str in xattr_template:
         try:
             options = RenderOptions(
-                none_str=_OSXPHOTOS_NONE_SENTINEL,
+                none_str=_OSXPHOTOS_NONE_SENTINEL, path_sep="/", export_dir=export_dir
-                path_sep="/",
-                export_dir=export_dir,
-                exportdb=export_db,
             )
             rendered, unmatched = photo.render_template(template_str, options)
         except ValueError as e:
@@ -3751,9 +3772,7 @@ def run_post_command(
             # some categories, like error, return a tuple of (file, error str)
             if isinstance(f, tuple):
                 f = f[0]
-            render_options = RenderOptions(
+            render_options = RenderOptions(export_dir=export_dir, filepath=f)
-                export_dir=export_dir, filepath=f, exportdb=export_db
-            )
             template = PhotoTemplate(photo, exiftool_path=exiftool_path)
             command, _ = template.render(command_template, options=render_options)
             command = command[0] if command else None

osxphotos/crash_reporter.py

@@ -0,0 +1,46 @@
+"""Error logger/crash reporter decorator"""
+
+import datetime
+import functools
+import platform
+import sys
+import traceback
+
+from rich import print
+
+
+def crash_reporter(filename, message, title, postamble, *extra_args):
+    """Create a crash dump file on error named filename
+
+    On error, create a crash dump file named filename with exception and stack trace.
+    message is printed to stderr
+    title is printed at beginning of crash dump file
+    postamble is printed to stderr after crash dump file is created
+    If extra_args is not None, any additional arguments to the function will be printed to the file.
+    """
+
+    def decorated(func):
+        @functools.wraps(func)
+        def wrapped(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except Exception as e:
+                print(message, file=sys.stderr)
+                print(f"[red]{e}[/red]", file=sys.stderr)
+                with open(filename, "w") as f:
+                    f.write(f"{title}\n")
+                    f.write(f"Created: {datetime.datetime.now()}\n")
+                    f.write(f"Python version: {sys.version}\n")
+                    f.write(f"Platform: {platform.platform()}\n")
+                    f.write(f"sys.argv: {sys.argv}\n")
+                    for arg in extra_args:
+                        f.write(f"{arg}\n")
+                    f.write(f"Error: {e}\n")
+                    traceback.print_exc(file=f)
+                print(f"Crash log written to '{filename}'", file=sys.stderr)
+                print(f"{postamble}", file=sys.stderr)
+                sys.exit(1)
+
+        return wrapped
+
+    return decorated

osxphotos/exiftool.py

@@ -11,6 +11,7 @@ import html
 import json
 import logging
 import os
+import pathlib
 import re
 import shutil
 import subprocess
@@ -19,11 +20,12 @@ from functools import lru_cache  # pylint: disable=syntax-error
 
 __all__ = [
     "escape_str",
-    "unescape_str",
+    "exiftool_can_write",
-    "terminate_exiftool",
-    "get_exiftool_path",
     "ExifTool",
     "ExifToolCaching",
+    "get_exiftool_path",
+    "terminate_exiftool",
+    "unescape_str",
 ]
 
 # exiftool -stay_open commands outputs this EOF marker after command is run
@@ -33,6 +35,24 @@ EXIFTOOL_STAYOPEN_EOF_LEN = len(EXIFTOOL_STAYOPEN_EOF)
 # list of exiftool processes to cleanup when exiting or when terminate is called
 EXIFTOOL_PROCESSES = []
 
+# exiftool supported file types, created by utils/exiftool_supported_types.py
+EXIFTOOL_FILETYPES_JSON = "exiftool_filetypes.json"
+with (pathlib.Path(__file__).parent / EXIFTOOL_FILETYPES_JSON).open("r") as f:
+    EXIFTOOL_SUPPORTED_FILETYPES = json.load(f)
+
+
+def exiftool_can_write(suffix: str) -> bool:
+    """Return True if exiftool supports writing to a file with the given suffix, otherwise False"""
+    if not suffix:
+        return False
+    suffix = suffix.lower()
+    if suffix[0] == ".":
+        suffix = suffix[1:]
+    return (
+        suffix in EXIFTOOL_SUPPORTED_FILETYPES
+        and EXIFTOOL_SUPPORTED_FILETYPES[suffix]["write"]
+    )
+
 
 def escape_str(s):
     """escape string for use with exiftool -E"""

osxphotos/export_db.py

@@ -10,13 +10,18 @@ import sys
 from abc import ABC, abstractmethod
 from io import StringIO
 from sqlite3 import Error
+from typing import Union
 
 from ._constants import OSXPHOTOS_EXPORT_DB
 from ._version import __version__
+from .utils import normalize_fs_path
 
 __all__ = ["ExportDB_ABC", "ExportDBNoOp", "ExportDB", "ExportDBInMemory"]
 
-OSXPHOTOS_EXPORTDB_VERSION = "4.0"
+OSXPHOTOS_EXPORTDB_VERSION = "5.0"
+OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH = "4.3"
+OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_TABLES = "4.3"
+
 OSXPHOTOS_ABOUT_STRING = f"Created by osxphotos version {__version__} (https://github.com/RhetTbull/osxphotos) on {datetime.datetime.now()}"
 
 
@@ -99,17 +104,26 @@ class ExportDB_ABC(ABC):
     def set_detected_text_for_uuid(self, uuid, json_text):
         pass
 
+    @abstractmethod
+    def set_metadata_for_file(self, filename, metadata):
+        pass
+
+    @abstractmethod
+    def get_metadata_for_file(self, filename):
+        pass
+
     @abstractmethod
     def set_data(
         self,
         filename,
         uuid,
-        orig_stat,
+        orig_stat=None,
-        exif_stat,
+        exif_stat=None,
-        converted_stat,
+        converted_stat=None,
-        edited_stat,
+        edited_stat=None,
-        info_json,
+        info_json=None,
-        exif_json,
+        exif_json=None,
+        metadata=None,
     ):
         pass
 
@@ -179,16 +193,23 @@ class ExportDBNoOp(ExportDB_ABC):
     def set_detected_text_for_uuid(self, uuid, json_text):
         pass
 
+    def set_metadata_for_file(self, filename, metadata):
+        pass
+
+    def get_metadata_for_file(self, filename):
+        pass
+
     def set_data(
         self,
         filename,
         uuid,
-        orig_stat,
+        orig_stat=None,
-        exif_stat,
+        exif_stat=None,
-        converted_stat,
+        converted_stat=None,
-        edited_stat,
+        edited_stat=None,
-        info_json,
+        info_json=None,
-        exif_json,
+        exif_json=None,
+        metadata=None,
     ):
         pass
 
@@ -211,12 +232,13 @@ class ExportDB(ExportDB_ABC):
         """query database for filename and return UUID
         returns None if filename not found in database
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filepath_normalized = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
             c.execute(
-                "SELECT uuid FROM files WHERE filepath_normalized = ?", (filename,)
+                "SELECT uuid FROM files WHERE filepath_normalized = ?",
+                (filepath_normalized,),
             )
             results = c.fetchone()
             uuid = results[0] if results else None
@@ -228,7 +250,7 @@ class ExportDB(ExportDB_ABC):
     def set_uuid_for_file(self, filename, uuid):
         """set UUID of filename to uuid in the database"""
         filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -245,7 +267,7 @@ class ExportDB(ExportDB_ABC):
         """set stat info for filename
         filename: filename to set the stat info for
         stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -266,7 +288,7 @@ class ExportDB(ExportDB_ABC):
         """get stat info for filename
         returns: tuple of (mode, size, mtime)
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -302,7 +324,7 @@ class ExportDB(ExportDB_ABC):
         """set stat info for filename (after exiftool has updated it)
         filename: filename to set the stat info for
         stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -323,7 +345,7 @@ class ExportDB(ExportDB_ABC):
         """get stat info for filename (after exiftool has updated it)
         returns: tuple of (mode, size, mtime)
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -384,7 +406,7 @@ class ExportDB(ExportDB_ABC):
 
     def get_exifdata_for_file(self, filename):
         """returns the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -402,7 +424,7 @@ class ExportDB(ExportDB_ABC):
 
     def set_exifdata_for_file(self, filename, exifdata):
         """sets the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -416,7 +438,7 @@ class ExportDB(ExportDB_ABC):
 
     def get_sidecar_for_file(self, filename):
         """returns the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -444,7 +466,7 @@ class ExportDB(ExportDB_ABC):
 
     def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
         """sets the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -502,56 +524,112 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
-    def set_data(
+    def set_metadata_for_file(self, filename, metadata):
-        self,
+        """set metadata of filename in the database"""
-        filename,
-        uuid,
-        orig_stat,
-        exif_stat,
-        converted_stat,
-        edited_stat,
-        info_json,
-        exif_json,
-    ):
-        """sets all the data for file and uuid at once"""
         filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
         conn = self._conn
         try:
             c = conn.cursor()
             c.execute(
-                "INSERT OR REPLACE INTO files(filepath, filepath_normalized, uuid) VALUES (?, ?, ?);",
+                "UPDATE files SET metadata = ? WHERE filepath_normalized = ?;",
+                (metadata, filename_normalized),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
+
+    def get_metadata_for_file(self, filename):
+        """get metadata value for file"""
+        filename = self._normalize_filepath_relative(filename)
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "SELECT metadata FROM files WHERE filepath_normalized = ?",
+                (filename,),
+            )
+            results = c.fetchone()
+            metadata = results[0] if results else None
+        except Error as e:
+            logging.warning(e)
+            metadata = None
+
+        return metadata
+
+    def set_data(
+        self,
+        filename,
+        uuid,
+        orig_stat=None,
+        exif_stat=None,
+        converted_stat=None,
+        edited_stat=None,
+        info_json=None,
+        exif_json=None,
+        metadata=None,
+    ):
+        """sets all the data for file and uuid at once; if any value is None, does not set it"""
+        filename = str(pathlib.Path(filename).relative_to(self._path))
+        filename_normalized = self._normalize_filepath(filename)
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            # update files table (if needed);
+            # this statement works around fact that there was no unique constraint on files.filepath_normalized
+            c.execute(
+                """INSERT OR IGNORE INTO files(filepath, filepath_normalized, uuid) VALUES (?, ?, ?);""",
                 (filename, filename_normalized, uuid),
             )
 
-            c.execute(
+            if orig_stat is not None:
-                "UPDATE files "
+                c.execute(
-                + "SET orig_mode = ?, orig_size = ?, orig_mtime = ? "
+                    "UPDATE files "
-                + "WHERE filepath_normalized = ?;",
+                    + "SET orig_mode = ?, orig_size = ?, orig_mtime = ? "
-                (*orig_stat, filename_normalized),
+                    + "WHERE filepath_normalized = ?;",
-            )
+                    (*orig_stat, filename_normalized),
-            c.execute(
+                )
-                "UPDATE files "
+
-                + "SET exif_mode = ?, exif_size = ?, exif_mtime = ? "
+            if exif_stat is not None:
-                + "WHERE filepath_normalized = ?;",
+                c.execute(
-                (*exif_stat, filename_normalized),
+                    "UPDATE files "
-            )
+                    + "SET exif_mode = ?, exif_size = ?, exif_mtime = ? "
-            c.execute(
+                    + "WHERE filepath_normalized = ?;",
-                "INSERT OR REPLACE INTO converted(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                    (*exif_stat, filename_normalized),
-                (filename_normalized, *converted_stat),
+                )
-            )
+
-            c.execute(
+            if converted_stat is not None:
-                "INSERT OR REPLACE INTO edited(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
+                c.execute(
-                (filename_normalized, *edited_stat),
+                    "INSERT OR REPLACE INTO converted(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
-            )
+                    (filename_normalized, *converted_stat),
-            c.execute(
+                )
-                "INSERT OR REPLACE INTO info(uuid, json_info) VALUES (?, ?);",
+
-                (uuid, info_json),
+            if edited_stat is not None:
-            )
+                c.execute(
-            c.execute(
+                    "INSERT OR REPLACE INTO edited(filepath_normalized, mode, size, mtime) VALUES (?, ?, ?, ?);",
-                "INSERT OR REPLACE INTO exifdata(filepath_normalized, json_exifdata) VALUES (?, ?);",
+                    (filename_normalized, *edited_stat),
-                (filename_normalized, exif_json),
+                )
-            )
+
+            if info_json is not None:
+                c.execute(
+                    "INSERT OR REPLACE INTO info(uuid, json_info) VALUES (?, ?);",
+                    (uuid, info_json),
+                )
+
+            if exif_json is not None:
+                c.execute(
+                    "INSERT OR REPLACE INTO exifdata(filepath_normalized, json_exifdata) VALUES (?, ?);",
+                    (filename_normalized, exif_json),
+                )
+
+            if metadata is not None:
+                c.execute(
+                    "UPDATE files "
+                    + "SET metadata = ? "
+                    + "WHERE filepath_normalized = ?;",
+                    (metadata, filename_normalized),
+                )
+
             conn.commit()
         except Error as e:
             logging.warning(e)
@@ -564,7 +642,7 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
 
     def _set_stat_for_file(self, table, filename, stats):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -577,7 +655,7 @@ class ExportDB(ExportDB_ABC):
         conn.commit()
 
     def _get_stat_for_file(self, table, filename):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         c = conn.cursor()
         c.execute(
@@ -604,7 +682,7 @@ class ExportDB(ExportDB_ABC):
             conn = self._get_db_connection(dbfile)
             if not conn:
                 raise Exception("Error getting connection to database {dbfile}")
-            self._create_db_tables(conn)
+            self._create_or_migrate_db_tables(conn)
             self.was_created = True
             self.was_upgraded = ()
         else:
@@ -612,11 +690,19 @@ class ExportDB(ExportDB_ABC):
             self.was_created = False
             version_info = self._get_database_version(conn)
             if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION:
-                self._create_db_tables(conn)
+                self._create_or_migrate_db_tables(conn)
                 self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()
         self.version = OSXPHOTOS_EXPORTDB_VERSION
+
+        # turn on performance optimizations
+        c = conn.cursor()
+        c.execute("PRAGMA journal_mode=WAL;")
+        c.execute("PRAGMA synchronous=NORMAL;")
+        c.execute("PRAGMA cache_size=-100000;")
+        c.execute("PRAGMA temp_store=MEMORY;")
+
         return conn
 
     def _get_db_connection(self, dbfile):
@@ -636,88 +722,97 @@ class ExportDB(ExportDB_ABC):
         ).fetchone()
         return (version_info[0], version_info[1])
 
-    def _create_db_tables(self, conn):
+    def _create_or_migrate_db_tables(self, conn):
-        """create (if not already created) the necessary db tables for the export database
+        """create (if not already created) the necessary db tables for the export database and apply any needed migrations
-        conn: sqlite3 db connection
+
+        Args:
+            conn: sqlite3 db connection
         """
-        sql_commands = {
+        try:
-            "sql_version_table": """ CREATE TABLE IF NOT EXISTS version (
+            version = self._get_database_version(conn)
-                                id INTEGER PRIMARY KEY,
+        except Exception as e:
-                                osxphotos TEXT,
+            version = (__version__, OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_TABLES)
-                                exportdb TEXT 
+
-                                ); """,
+        # Current for version 4.3, for anything greater, do a migration after creation
-            "sql_about_table": """ CREATE TABLE IF NOT EXISTS about (
+        sql_commands = [
-                                id INTEGER PRIMARY KEY,
+            """ CREATE TABLE IF NOT EXISTS version (
-                                about TEXT
+                    id INTEGER PRIMARY KEY,
-                                );""",
+                    osxphotos TEXT,
-            "sql_files_table": """ CREATE TABLE IF NOT EXISTS files (
+                    exportdb TEXT 
-                              id INTEGER PRIMARY KEY,
+                    ); """,
-                              filepath TEXT NOT NULL,
+            """ CREATE TABLE IF NOT EXISTS about (
-                              filepath_normalized TEXT NOT NULL,
+                    id INTEGER PRIMARY KEY,
-                              uuid TEXT,
+                    about TEXT
-                              orig_mode INTEGER,
+                    );""",
-                              orig_size INTEGER,
+            """ CREATE TABLE IF NOT EXISTS files (
-                              orig_mtime REAL,
+                    id INTEGER PRIMARY KEY,
-                              exif_mode INTEGER,
+                    filepath TEXT NOT NULL,
-                              exif_size INTEGER,
+                    filepath_normalized TEXT NOT NULL,
-                              exif_mtime REAL
+                    uuid TEXT,
-                              ); """,
+                    orig_mode INTEGER,
-            "sql_runs_table": """ CREATE TABLE IF NOT EXISTS runs (
+                    orig_size INTEGER,
-                             id INTEGER PRIMARY KEY,
+                    orig_mtime REAL,
-                             datetime TEXT,
+                    exif_mode INTEGER,
-                             python_path TEXT,
+                    exif_size INTEGER,
-                             script_name TEXT,
+                    exif_mtime REAL
-                             args TEXT,
+                    ); """,
-                             cwd TEXT 
+            """ CREATE TABLE IF NOT EXISTS runs (
-                             ); """,
+                    id INTEGER PRIMARY KEY,
-            "sql_info_table": """ CREATE TABLE IF NOT EXISTS info (
+                    datetime TEXT,
-                             id INTEGER PRIMARY KEY,
+                    python_path TEXT,
-                             uuid text NOT NULL,
+                    script_name TEXT,
-                             json_info JSON 
+                    args TEXT,
-                             ); """,
+                    cwd TEXT 
-            "sql_exifdata_table": """ CREATE TABLE IF NOT EXISTS exifdata (
+                    ); """,
-                             id INTEGER PRIMARY KEY,
+            """ CREATE TABLE IF NOT EXISTS info (
-                             filepath_normalized TEXT NOT NULL,
+                    id INTEGER PRIMARY KEY,
-                             json_exifdata JSON 
+                    uuid text NOT NULL,
-                             ); """,
+                    json_info JSON 
-            "sql_edited_table": """ CREATE TABLE IF NOT EXISTS edited (
+                    ); """,
-                              id INTEGER PRIMARY KEY,
+            """ CREATE TABLE IF NOT EXISTS exifdata (
-                              filepath_normalized TEXT NOT NULL,
+                    id INTEGER PRIMARY KEY,
-                              mode INTEGER,
+                    filepath_normalized TEXT NOT NULL,
-                              size INTEGER,
+                    json_exifdata JSON 
-                              mtime REAL
+                    ); """,
-                              ); """,
+            """ CREATE TABLE IF NOT EXISTS edited (
-            "sql_converted_table": """ CREATE TABLE IF NOT EXISTS converted (
+                    id INTEGER PRIMARY KEY,
-                              id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
-                              filepath_normalized TEXT NOT NULL,
+                    mode INTEGER,
-                              mode INTEGER,
+                    size INTEGER,
-                              size INTEGER,
+                    mtime REAL
-                              mtime REAL
+                    ); """,
-                              ); """,
+            """ CREATE TABLE IF NOT EXISTS converted (
-            "sql_sidecar_table": """ CREATE TABLE IF NOT EXISTS sidecar (
+                    id INTEGER PRIMARY KEY,
-                              id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
-                              filepath_normalized TEXT NOT NULL,
+                    mode INTEGER,
-                              sidecar_data TEXT,
+                    size INTEGER,
-                              mode INTEGER,
+                    mtime REAL
-                              size INTEGER,
+                    ); """,
-                              mtime REAL
+            """ CREATE TABLE IF NOT EXISTS sidecar (
-                              ); """,
+                    id INTEGER PRIMARY KEY,
-            "sql_detected_text_table": """ CREATE TABLE IF NOT EXISTS detected_text (
+                    filepath_normalized TEXT NOT NULL,
-                              id INTEGER PRIMARY KEY,
+                    sidecar_data TEXT,
-                              uuid TEXT NOT NULL,
+                    mode INTEGER,
-                              text_data JSON
+                    size INTEGER,
-                              ); """,
+                    mtime REAL
-            "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
+                    ); """,
-            "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
+            """ CREATE TABLE IF NOT EXISTS detected_text (
-            "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
+                    id INTEGER PRIMARY KEY,
-            "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
+                    uuid TEXT NOT NULL,
-            "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+                    text_data JSON
-            "sql_sidecar_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
+                    ); """,
-            "sql_detected_text_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_detected_text on detected_text (uuid);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
-        }
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_detected_text on detected_text (uuid);""",
+        ]
+        # create the tables if needed
         try:
             c = conn.cursor()
-            for cmd in sql_commands.values():
+            for cmd in sql_commands:
                 c.execute(cmd)
             c.execute(
                 "INSERT INTO version(osxphotos, exportdb) VALUES (?, ?);",
@@ -728,6 +823,19 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+        # perform needed migrations
+        if version[1] < OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH:
+            self._migrate_normalized_filepath(conn)
+
+        if version[1] < OSXPHOTOS_EXPORTDB_VERSION:
+            try:
+                c = conn.cursor()
+                # add metadata column to files to support --force-update
+                c.execute("ALTER TABLE files ADD COLUMN metadata TEXT;")
+                conn.commit()
+            except Error as e:
+                logging.warning(e)
+
     def __del__(self):
         """ensure the database connection is closed"""
         try:
@@ -753,6 +861,54 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def _normalize_filepath(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, lower case"""
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _normalize_filepath_relative(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, relative path (to export dir), lower case"""
+        filepath = str(pathlib.Path(filepath).relative_to(self._path))
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _migrate_normalized_filepath(self, conn):
+        """Fix all filepath_normalized columns for unicode normalization"""
+        # Prior to database version 4.3, filepath_normalized was not normalized for unicode
+        c = conn.cursor()
+        migration_sql = [
+            """ CREATE TABLE IF NOT EXISTS files_migrate (
+                    id INTEGER PRIMARY KEY,
+                    filepath TEXT NOT NULL,
+                    filepath_normalized TEXT NOT NULL,
+                    uuid TEXT,
+                    orig_mode INTEGER,
+                    orig_size INTEGER,
+                    orig_mtime REAL,
+                    exif_mode INTEGER,
+                    exif_size INTEGER,
+                    exif_mtime REAL,
+                    UNIQUE(filepath_normalized)
+                    ); """,
+            """ INSERT INTO files_migrate SELECT * FROM files;""",
+            """ DROP TABLE files;""",
+            """ ALTER TABLE files_migrate RENAME TO files;""",
+        ]
+        for sql in migration_sql:
+            c.execute(sql)
+        conn.commit()
+
+        for table in ["converted", "edited", "exifdata", "files", "sidecar"]:
+            old_values = c.execute(
+                f"SELECT filepath_normalized, id FROM {table}"
+            ).fetchall()
+            new_values = [
+                (self._normalize_filepath(filepath_normalized), id_)
+                for filepath_normalized, id_ in old_values
+            ]
+            c.executemany(
+                f"UPDATE {table} SET filepath_normalized=? WHERE id=?", new_values
+            )
+        conn.commit()
+
 
 class ExportDBInMemory(ExportDB):
     """In memory version of ExportDB
@@ -778,7 +934,7 @@ class ExportDBInMemory(ExportDB):
             conn = self._get_db_connection()
             if not conn:
                 raise Exception("Error getting connection to in-memory database")
-            self._create_db_tables(conn)
+            self._create_or_migrate_db_tables(conn)
             self.was_created = True
             self.was_upgraded = ()
         else:
@@ -801,7 +957,7 @@ class ExportDBInMemory(ExportDB):
             self.was_created = False
             _, exportdb_ver = self._get_database_version(conn)
             if exportdb_ver < OSXPHOTOS_EXPORTDB_VERSION:
-                self._create_db_tables(conn)
+                self._create_or_migrate_db_tables(conn)
                 self.was_upgraded = (exportdb_ver, OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()

osxphotos/fileutil.py

@@ -181,7 +181,6 @@ class FileUtilMacOS(FileUtilABC):
             return False
 
         s1 = cls._sig(os.stat(f1))
-
         if s1[0] != stat.S_IFREG or s2[0] != stat.S_IFREG:
             return False
         return s1 == s2

osxphotos/photoinfo.py

@@ -35,6 +35,9 @@ from ._constants import (
     BURST_KEY,
     BURST_NOT_SELECTED,
     BURST_SELECTED,
+    SIDECAR_EXIFTOOL,
+    SIDECAR_JSON,
+    SIDECAR_XMP,
     TEXT_DETECTION_CONFIDENCE_THRESHOLD,
 )
 from .adjustmentsinfo import AdjustmentsInfo
@@ -43,7 +46,7 @@ from .exifinfo import ExifInfo
 from .exiftool import ExifToolCaching, get_exiftool_path
 from .momentinfo import MomentInfo
 from .personinfo import FaceInfo, PersonInfo
-from .photoexporter import PhotoExporter
+from .photoexporter import ExportOptions, PhotoExporter
 from .phototemplate import PhotoTemplate, RenderOptions
 from .placeinfo import PlaceInfo4, PlaceInfo5
 from .query_builder import get_query
@@ -51,7 +54,7 @@ from .scoreinfo import ScoreInfo
 from .searchinfo import SearchInfo
 from .text_detection import detect_text
 from .uti import get_preferred_uti_extension, get_uti_for_extension
-from .utils import _debug, _get_resource_loc, findfiles
+from .utils import _debug, _get_resource_loc, list_directory
 
 __all__ = ["PhotoInfo", "PhotoInfoNone"]
 
@@ -366,7 +369,7 @@ class PhotoInfo:
         # In Photos 5, raw is in same folder as original but with _4.ext
         # Unless "Copy Items to the Photos Library" is not checked
         # then RAW image is not renamed but has same name is jpeg buth with raw extension
-        # Current implementation uses findfiles to find images with the correct raw UTI extension
+        # Current implementation finds images with the correct raw UTI extension
         # in same folder as the original and with same stem as original in form: original_stem*.raw_ext
         # TODO: I don't like this -- would prefer a more deterministic approach but until I have more
         # data on how Photos stores and retrieves RAW images, this seems to be working
@@ -402,8 +405,7 @@ class PhotoInfo:
             # raw files have same name as original but with _4.raw_ext appended
             # I believe the _4 maps to PHAssetResourceTypeAlternatePhoto = 4
             # see: https://developer.apple.com/documentation/photokit/phassetresourcetype/phassetresourcetypealternatephoto?language=objc
-            glob_str = f"{filestem}_4*"
+            raw_file = list_directory(filepath, startswith=f"{filestem}_4")
-            raw_file = findfiles(glob_str, filepath)
             if not raw_file:
                 photopath = None
             else:
@@ -1490,28 +1492,48 @@ class PhotoInfo:
         """
 
         exporter = PhotoExporter(self)
-        return exporter.export(
+        sidecar = 0
-            dest=dest,
+        if sidecar_json:
-            filename=filename,
+            sidecar |= SIDECAR_JSON
+        if sidecar_exiftool:
+            sidecar |= SIDECAR_EXIFTOOL
+        if sidecar_xmp:
+            sidecar |= SIDECAR_XMP
+
+        if not filename:
+            if not edited:
+                filename = self.original_filename
+            else:
+                original_name = pathlib.Path(self.original_filename)
+                if self.path_edited:
+                    ext = pathlib.Path(self.path_edited).suffix
+                else:
+                    uti = self.uti_edited if edited and self.uti_edited else self.uti
+                    ext = get_preferred_uti_extension(uti)
+                    ext = "." + ext
+                filename = original_name.stem + "_edited" + ext
+
+        options = ExportOptions(
+            description_template=description_template,
             edited=edited,
-            live_photo=live_photo,
-            raw_photo=raw_photo,
-            export_as_hardlink=export_as_hardlink,
-            overwrite=overwrite,
-            increment=increment,
-            sidecar_json=sidecar_json,
-            sidecar_exiftool=sidecar_exiftool,
-            sidecar_xmp=sidecar_xmp,
-            use_photos_export=use_photos_export,
-            timeout=timeout,
             exiftool=exiftool,
+            export_as_hardlink=export_as_hardlink,
+            increment=increment,
+            keyword_template=keyword_template,
+            live_photo=live_photo,
+            overwrite=overwrite,
+            raw_photo=raw_photo,
+            render_options=render_options,
+            sidecar=sidecar,
+            timeout=timeout,
             use_albums_as_keywords=use_albums_as_keywords,
             use_persons_as_keywords=use_persons_as_keywords,
-            keyword_template=keyword_template,
+            use_photos_export=use_photos_export,
-            description_template=description_template,
-            render_options=render_options,
         )
 
+        results = exporter.export(dest, filename=filename, options=options)
+        return results.exported
+
     def _get_album_uuids(self, project=False):
         """Return list of album UUIDs this photo is found in
 
@@ -1706,7 +1728,11 @@ class PhotoInfo:
             if isinstance(o, (datetime.date, datetime.datetime)):
                 return o.isoformat()
 
-        return json.dumps(self.asdict(), sort_keys=True, default=default)
+        dict_data = self.asdict()
+        for k, v in dict_data.items():
+            if v and isinstance(v, (list, tuple)) and not isinstance(v[0], dict):
+                dict_data[k] = sorted(v)
+        return json.dumps(dict_data, sort_keys=True, default=default)
 
     def __eq__(self, other):
         """Compare two PhotoInfo objects for equality"""

osxphotos/photosdb/_photosdb_process_exif.py

@@ -4,7 +4,7 @@
 import logging
 
 from .._constants import _DB_TABLE_NAMES, _PHOTOS_4_VERSION
-from ..utils import _db_is_locked, _debug, _open_sql_file
+from ..utils import _db_is_locked, _open_sql_file
 from .photosdb_utils import get_db_version
 
 

osxphotos/photosdb/_photosdb_process_searchinfo.py

@@ -10,7 +10,7 @@ import uuid as uuidlib
 from pprint import pformat
 
 from .._constants import _PHOTOS_4_VERSION, SEARCH_CATEGORY_LABEL
-from ..utils import _db_is_locked, _debug, _open_sql_file, normalize_unicode
+from ..utils import _db_is_locked, _open_sql_file, normalize_unicode
 
 """
     This module should be imported in the class defintion of PhotosDB in photosdb.py
@@ -139,17 +139,6 @@ def _process_searchinfo(self):
                 _db_searchinfo_labels[label] = [uuid]
                 _db_searchinfo_labels_normalized[label_norm] = [uuid]
 
-    if _debug():
-        logging.debug(
-            "_db_searchinfo_categories: \n" + pformat(self._db_searchinfo_categories)
-        )
-        logging.debug("_db_searchinfo_uuid: \n" + pformat(self._db_searchinfo_uuid))
-        logging.debug("_db_searchinfo_labels: \n" + pformat(self._db_searchinfo_labels))
-        logging.debug(
-            "_db_searchinfo_labels_normalized: \n"
-            + pformat(self._db_searchinfo_labels_normalized)
-        )
-
     conn.close()
 
 

osxphotos/photosdb/photosdb.py

@@ -39,6 +39,7 @@ from .._constants import (
     _PHOTOS_5_PROJECT_ALBUM_KIND,
     _PHOTOS_5_ROOT_FOLDER_KIND,
     _PHOTOS_5_SHARED_ALBUM_KIND,
+    _PHOTOS_5_VERSION,
     _TESTED_OS_VERSIONS,
     _UNKNOWN_PERSON,
     BURST_KEY,
@@ -659,14 +660,18 @@ class PhotosDB:
 
         for person in c:
             pk = person[0]
-            fullname = person[2] if person[2] is not None else _UNKNOWN_PERSON
+            fullname = (
+                normalize_unicode(person[2])
+                if person[2] is not None
+                else _UNKNOWN_PERSON
+            )
             self._dbpersons_pk[pk] = {
                 "pk": pk,
                 "uuid": person[1],
                 "fullname": fullname,
                 "facecount": person[3],
                 "keyface": person[5],
-                "displayname": person[4],
+                "displayname": normalize_unicode(person[4]),
                 "photo_uuid": None,
                 "keyface_uuid": None,
             }
@@ -733,13 +738,6 @@ class PhotosDB:
             except KeyError:
                 self._dbfaces_pk[pk] = [uuid]
 
-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
         # Get info on albums
         verbose("Processing albums.")
         c.execute(
@@ -876,14 +874,6 @@ class PhotosDB:
             else:
                 self._dbalbum_folders[album] = {}
 
-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-            logging.debug(pformat(self._dbfolder_details))
-
         # Get info on keywords
         verbose("Processing keywords.")
         c.execute(
@@ -899,13 +889,16 @@ class PhotosDB:
                 RKMaster.uuid = RKVersion.masterUuid
             """
         )
-        for keyword in c:
+        for keyword_title, keyword_uuid, _ in c:
-            if not keyword[1] in self._dbkeywords_uuid:
+            keyword_title = normalize_unicode(keyword_title)
-                self._dbkeywords_uuid[keyword[1]] = []
+            try:
-            if not keyword[0] in self._dbkeywords_keyword:
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
-                self._dbkeywords_keyword[keyword[0]] = []
+            except KeyError:
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
-            self._dbkeywords_keyword[keyword[0]].append(keyword[1])
+            try:
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
+            except KeyError:
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]
 
         # Get info on disk volumes
         c.execute("select RKVolume.modelId, RKVolume.name from RKVolume")
@@ -1027,13 +1020,11 @@ class PhotosDB:
 
         for row in c:
             uuid = row[0]
-            if _debug():
-                logging.debug(f"uuid = '{uuid}, master = '{row[2]}")
             self._dbphotos[uuid] = {}
             self._dbphotos[uuid]["_uuid"] = uuid  # stored here for easier debugging
             self._dbphotos[uuid]["modelID"] = row[1]
             self._dbphotos[uuid]["masterUuid"] = row[2]
-            self._dbphotos[uuid]["filename"] = row[3]
+            self._dbphotos[uuid]["filename"] = normalize_unicode(row[3])
 
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
@@ -1272,13 +1263,13 @@ class PhotosDB:
             info["volumeId"] = row[1]
             info["imagePath"] = row[2]
             info["isMissing"] = row[3]
-            info["originalFilename"] = row[4]
+            info["originalFilename"] = normalize_unicode(row[4])
             info["UTI"] = row[5]
             info["modelID"] = row[6]
             info["fileSize"] = row[7]
             info["isTrulyRAW"] = row[8]
             info["alternateMasterUuid"] = row[9]
-            info["filename"] = row[10]
+            info["filename"] = normalize_unicode(row[10])
             self._dbphotos_master[uuid] = info
 
         # get details needed to find path of the edited photos
@@ -1550,39 +1541,6 @@ class PhotosDB:
 
         # done processing, dump debug data if requested
         verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))
 
     def _build_album_folder_hierarchy_4(self, uuid, folders=None):
         """recursively build folder/album hierarchy
@@ -1673,7 +1631,7 @@ class PhotosDB:
         for person in c:
             pk = person[0]
             fullname = (
-                person[2]
+                normalize_unicode(person[2])
                 if (person[2] != "" and person[2] is not None)
                 else _UNKNOWN_PERSON
             )
@@ -1683,7 +1641,7 @@ class PhotosDB:
                 "fullname": fullname,
                 "facecount": person[3],
                 "keyface": person[4],
-                "displayname": person[5],
+                "displayname": normalize_unicode(person[5]),
                 "photo_uuid": None,
                 "keyface_uuid": None,
             }
@@ -1747,13 +1705,6 @@ class PhotosDB:
             except KeyError:
                 self._dbfaces_pk[pk] = [uuid]
 
-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
         # get details about albums
         verbose("Processing albums.")
         c.execute(
@@ -1870,13 +1821,6 @@ class PhotosDB:
                 # shared albums can't be in folders
                 self._dbalbum_folders[album] = []
 
-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-
         # get details on keywords
         verbose("Processing keywords.")
         c.execute(
@@ -1886,29 +1830,22 @@ class PhotosDB:
                 JOIN Z_1KEYWORDS ON Z_1KEYWORDS.Z_1ASSETATTRIBUTES = ZADDITIONALASSETATTRIBUTES.Z_PK 
                 JOIN ZKEYWORD ON ZKEYWORD.Z_PK = {keyword_join} """
         )
-        for keyword in c:
+        for keyword_title, keyword_uuid in c:
-            keyword_title = normalize_unicode(keyword[0])
+            keyword_title = normalize_unicode(keyword_title)
-            if not keyword[1] in self._dbkeywords_uuid:
+            try:
-                self._dbkeywords_uuid[keyword[1]] = []
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
-            if not keyword_title in self._dbkeywords_keyword:
+            except KeyError:
-                self._dbkeywords_keyword[keyword_title] = []
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
+            try:
-            self._dbkeywords_keyword[keyword_title].append(keyword[1])
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
-
+            except KeyError:
-        if _debug():
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]
-            logging.debug(f"Finished walking through keywords")
-            logging.debug(pformat(self._dbkeywords_keyword))
-            logging.debug(pformat(self._dbkeywords_uuid))
 
         # get details on disk volumes
         c.execute("SELECT ZUUID, ZNAME from ZFILESYSTEMVOLUME")
         for vol in c:
             self._dbvolumes[vol[0]] = vol[1]
 
-        if _debug():
-            logging.debug(f"Finished walking through volumes")
-            logging.debug(self._dbvolumes)
-
         # get details about photos
         verbose("Processing photo details.")
         c.execute(
@@ -2042,8 +1979,8 @@ class PhotosDB:
 
             info["hidden"] = row[9]
             info["favorite"] = row[10]
-            info["originalFilename"] = row[3]
+            info["originalFilename"] = normalize_unicode(row[3])
-            info["filename"] = row[12]
+            info["filename"] = normalize_unicode(row[12])
             info["directory"] = row[11]
 
             # set latitude and longitude
@@ -2519,50 +2456,7 @@ class PhotosDB:
         verbose("Processing moments.")
         self._process_moments()
 
-        # done processing, dump debug data if requested
         verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Album folders (_dbalbum_folders):")
-            logging.debug(pformat(self._dbalbum_folders))
-
-            logging.debug("Album parent folders (_dbalbum_parent_folders):")
-            logging.debug(pformat(self._dbalbum_parent_folders))
-
-            logging.debug("Albums pk (_dbalbums_pk):")
-            logging.debug(pformat(self._dbalbums_pk))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))
 
     def _process_moments(self):
         """Process data from ZMOMENT table"""
@@ -2623,8 +2517,8 @@ class PhotosDB:
             moment_info["modificationDate"] = row[6]
             moment_info["representativeDate"] = row[7]
             moment_info["startDate"] = row[8]
-            moment_info["subtitle"] = row[9]
+            moment_info["subtitle"] = normalize_unicode(row[9])
-            moment_info["title"] = row[10]
+            moment_info["title"] = normalize_unicode(row[10])
             moment_info["uuid"] = row[11]
 
             # if both lat/lon == -180, then it means location undefined
@@ -3027,6 +2921,7 @@ class PhotosDB:
             if keywords:
                 keyword_set = set()
                 for keyword in keywords:
+                    keyword = normalize_unicode(keyword)
                     if keyword in self._dbkeywords_keyword:
                         keyword_set.update(self._dbkeywords_keyword[keyword])
                 photos_sets.append(keyword_set)
@@ -3034,6 +2929,7 @@ class PhotosDB:
             if persons:
                 person_set = set()
                 for person in persons:
+                    person = normalize_unicode(person)
                     if person in self._dbpersons_fullname:
                         for pk in self._dbpersons_fullname[person]:
                             try:
@@ -3076,8 +2972,6 @@ class PhotosDB:
                 ):
                     info = PhotoInfo(db=self, uuid=p, info=self._dbphotos[p])
                     photoinfo.append(info)
-        if _debug:
-            logging.debug(f"photoinfo: {pformat(photoinfo)}")
 
         return photoinfo
 
@@ -3414,23 +3308,35 @@ class PhotosDB:
                 # case-insensitive
                 for n in name:
                     n = n.lower()
-                    photo_list.extend(
+                    if self._db_version >= _PHOTOS_5_VERSION:
-                        [
+                        # search only original_filename (#594)
-                            p
+                        photo_list.extend(
-                            for p in photos
+                            [p for p in photos if n in p.original_filename.lower()]
-                            if n in p.filename.lower()
+                        )
-                            or n in p.original_filename.lower()
+                    else:
-                        ]
+                        photo_list.extend(
-                    )
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename.lower()
+                                or n in p.original_filename.lower()
+                            ]
+                        )
             else:
                 for n in name:
-                    photo_list.extend(
+                    if self._db_version >= _PHOTOS_5_VERSION:
-                        [
+                        # search only original_filename (#594)
-                            p
+                        photo_list.extend(
-                            for p in photos
+                            [p for p in photos if n in p.original_filename]
-                            if n in p.filename or n in p.original_filename
+                        )
-                        ]
+                    else:
-                    )
+                        photo_list.extend(
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename or n in p.original_filename
+                            ]
+                        )
             photos = photo_list
 
         if options.min_size:

osxphotos/phototemplate.py

@@ -17,7 +17,6 @@ from ._constants import _UNKNOWN_PERSON, TEXT_DETECTION_CONFIDENCE_THRESHOLD
 from ._version import __version__
 from .datetime_formatter import DateTimeFormatter
 from .exiftool import ExifToolCaching
-from .export_db import ExportDB_ABC, ExportDBInMemory
 from .path_utils import sanitize_dirname, sanitize_filename, sanitize_pathpart
 from .text_detection import detect_text
 from .utils import expand_and_validate_filepath, load_function
@@ -300,7 +299,6 @@ class RenderOptions:
     dest_path: set to the destination path of the photo (for use by {function} template), only valid with --filename
     filepath: set to value for filepath of the exported photo if you want to evaluate {filepath} template
     quote: quote path templates for execution in the shell
-    exportdb: ExportDB object
     """
 
     none_str: str = "_"
@@ -315,7 +313,6 @@ class RenderOptions:
     dest_path: Optional[str] = None
     filepath: Optional[str] = None
     quote: bool = False
-    exportdb: Optional[ExportDB_ABC] = None
 
 
 class PhotoTemplateParser:
@@ -384,9 +381,6 @@ class PhotoTemplate:
         self.filepath = options.filepath
         self.quote = options.quote
         self.dest_path = options.dest_path
-        self.exportdb = options.exportdb or ExportDBInMemory(
-            None, self.export_dir or "."
-        )
 
     def render(
         self,
@@ -420,7 +414,6 @@ class PhotoTemplate:
         self.filepath = options.filepath
         self.quote = options.quote
         self.dest_path = options.dest_path
-        self.exportdb = options.exportdb or self.exportdb
 
         try:
             model = self.parser.parse(template)
@@ -1216,7 +1209,7 @@ class PhotoTemplate:
             else:
                 values = list(obj)
         elif field == "detected_text":
-            values = _get_detected_text(self.photo, self.exportdb, confidence=subfield)
+            values = _get_detected_text(self.photo, confidence=subfield)
         else:
             raise ValueError(f"Unhandled template value: {field}")
 
@@ -1459,7 +1452,7 @@ def _get_album_by_path(photo, folder_album_path):
     return None
 
 
-def _get_detected_text(photo, exportdb, confidence=TEXT_DETECTION_CONFIDENCE_THRESHOLD):
+def _get_detected_text(photo, confidence=TEXT_DETECTION_CONFIDENCE_THRESHOLD):
     """Returns the detected text for a photo
     {detected_text} uses this instead of PhotoInfo.detected_text() to cache the text for all confidence values
     """
@@ -1475,5 +1468,4 @@ def _get_detected_text(photo, exportdb, confidence=TEXT_DETECTION_CONFIDENCE_THR
     # _detected_text caches the text detection results in an extended attribute
     # so the first time this gets called is slow but repeated accesses are fast
     detected_text = photo._detected_text()
-    exportdb.set_detected_text_for_uuid(photo.uuid, json.dumps(detected_text))
     return [text for text, conf in detected_text if conf >= confidence]

osxphotos/phototemplate.tx

@@ -99,7 +99,7 @@ OPERATOR:
 PathSep:
     (
     "("
-    (value=/[^\(\)\{\}]{0,1}/)?
+    (value=/[^\(\)\{\}]+/)?
     ")"
     )?
 ;

osxphotos/searchinfo.py

@@ -211,10 +211,12 @@ class SearchInfo:
         """return list of text for a specified category ID"""
         if self._db_searchinfo:
             content = "normalized_string" if self._normalized else "content_string"
-            return [
+            return sorted(
-                rec[content]
+                [
-                for rec in self._db_searchinfo
+                    rec[content]
-                if rec["category"] == category
+                    for rec in self._db_searchinfo
-            ]
+                    if rec["category"] == category
+                ]
+            )
         else:
             return []

osxphotos/templates/xmp_sidecar.mako

@@ -103,6 +103,8 @@
     % if photo.face_info:
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>${photo.width if photo.orientation in [5, 6, 7, 8] else photo.height}</stDim:h>
+    <stDim:w>${photo.height if photo.orientation in [5, 6, 7, 8] else photo.width}</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

osxphotos/templates/xmp_sidecar_beta.mako

@@ -103,6 +103,8 @@
     % if photo.face_info:
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>${photo.width if photo.orientation in [5, 6, 7, 8] else photo.height}</stDim:h>
+    <stDim:w>${photo.height if photo.orientation in [5, 6, 7, 8] else photo.width}</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

osxphotos/utils.py

@@ -1,5 +1,6 @@
 """ Utility functions used in osxphotos """
 
+import datetime
 import fnmatch
 import glob
 import importlib
@@ -16,18 +17,17 @@ import sys
 import unicodedata
 import urllib.parse
 from plistlib import load as plistload
-from typing import Callable, List, Union
+from typing import Callable, List, Union, Optional
 
 import CoreFoundation
 import objc
-from Foundation import NSFileManager, NSString
+from Foundation import NSFileManager, NSPredicate, NSString
 
 from ._constants import UNICODE_FORMAT
 
 __all__ = [
     "dd_to_dms_str",
     "expand_and_validate_filepath",
-    "findfiles",
     "get_last_library_path",
     "get_system_library_path",
     "increment_filename_with_count",
@@ -265,7 +265,9 @@ def list_photo_libraries():
     # On older MacOS versions, mdfind appears to ignore some libraries
     # glob to find libraries in ~/Pictures then mdfind to find all the others
     # TODO: make this more robust
-    lib_list = glob.glob(f"{str(pathlib.Path.home())}/Pictures/*.photoslibrary")
+    lib_list = list_directory(
+        f"{pathlib.Path.home()}/Pictures/", glob="*.photoslibrary"
+    )
 
     # On older OS, may not get all libraries so make sure we get the last one
     last_lib = get_last_library_path()
@@ -284,35 +286,95 @@ def list_photo_libraries():
 
 def normalize_fs_path(path: str) -> str:
     """Normalize filesystem paths with unicode in them"""
-    with objc.autorelease_pool():
+    # macOS HFS+ uses NFD, APFS doesn't normalize but stick with NFD
-        normalized_path = NSString.fileSystemRepresentation(path)
+    # ref: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/
-        return normalized_path.decode("utf8")
+    return unicodedata.normalize("NFD", path)
 
 
-def findfiles(pattern, path_):
+# def findfiles(pattern, path):
-    """Returns list of filenames from path_ matched by pattern
+#     """Returns list of filenames from path matched by pattern
-    shell pattern. Matching is case-insensitive.
+#     shell pattern. Matching is case-insensitive.
-    If 'path_' is invalid/doesn't exist, returns []."""
+#     If 'path_' is invalid/doesn't exist, returns []."""
-    if not os.path.isdir(path_):
+#     if not os.path.isdir(path):
+#         return []
+
+#     # paths need to be normalized for unicode as filesystem returns unicode in NFD form
+#     pattern = normalize_fs_path(pattern)
+#     rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
+#     files = os.listdir(path)
+#     return [name for name in files if rule.match(name)]
+
+
+def list_directory(
+    directory: Union[str, pathlib.Path],
+    startswith: Optional[str] = None,
+    endswith: Optional[str] = None,
+    contains: Optional[str] = None,
+    glob: Optional[str] = None,
+    include_path: bool = False,
+    case_sensitive: bool = False,
+) -> List[Union[str, pathlib.Path]]:
+    """List directory contents and return list of files or directories matching search criteria.
+    Accounts for case-insensitive filesystems, unicode filenames. directory can be a str or a pathlib.Path object.
+
+    Args:
+        directory: directory to search
+        startswith: string to match at start of filename
+        endswith: string to match at end of filename
+        contains: string to match anywhere in filename
+        glob: shell-style glob pattern to match filename
+        include_path: if True, return full path to file
+        case_sensitive: if True, match case-sensitively
+
+    Returns: List of files or directories matching search criteria as either str or pathlib.Path objects depending on the input type;
+    returns empty list if directory is invalid or doesn't exist.
+
+    """
+    is_pathlib = isinstance(directory, pathlib.Path)
+    if is_pathlib:
+        directory = str(directory)
+
+    if not os.path.isdir(directory):
         return []
-    # See: https://gist.github.com/techtonik/5694830
 
-    # paths need to be normalized for unicode as filesystem returns unicode in NFD form
+    startswith = normalize_fs_path(startswith) if startswith else None
-    pattern = normalize_fs_path(pattern)
+    endswith = normalize_fs_path(endswith) if endswith else None
-    rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
+    contains = normalize_fs_path(contains) if contains else None
-    files = list_directory(path_)
+    glob = normalize_fs_path(glob) if glob else None
-    return [name for name in files if rule.match(name)]
 
+    files = [normalize_fs_path(f) for f in os.listdir(directory)]
+    if not case_sensitive:
+        files_normalized = {f.lower(): f for f in files}
+        files = [f.lower() for f in files]
+        startswith = startswith.lower() if startswith else None
+        endswith = endswith.lower() if endswith else None
+        contains = contains.lower() if contains else None
+        glob = glob.lower() if glob else None
+    else:
+        files_normalized = {f: f for f in files}
 
-def list_directory(directory_path: str) -> List[str]:
+    if startswith:
-    """List directory contents using NSFileManager"""
+        files = [f for f in files if f.startswith(startswith)]
-    """[[NSFileManager defaultManager] contentsOfDirectoryAtPath:@"directoryName" error:nil]"""
+    if endswith:
-    with objc.autorelease_pool():
+        endswith = normalize_fs_path(endswith)
-        manager = NSFileManager.defaultManager()
+        files = [f for f in files if f.endswith(endswith)]
-        contents, error = manager.contentsOfDirectoryAtPath_error_(directory_path, None)
+    if contains:
-        if error:
+        contains = normalize_fs_path(contains)
-            raise OSError(f"Error listing directory {directory_path}: {error}")
+        files = [f for f in files if contains in f]
-        return [str(path) for path in contents]
+    if glob:
+        glob = normalize_fs_path(glob)
+        flags = re.IGNORECASE if not case_sensitive else 0
+        rule = re.compile(fnmatch.translate(glob), flags)
+        files = [f for f in files if rule.match(f)]
+
+    files = [files_normalized[f] for f in files]
+
+    if include_path:
+        files = [os.path.join(directory, f) for f in files]
+    if is_pathlib:
+        files = [pathlib.Path(f) for f in files]
+
+    return files
 
 
 def _open_sql_file(dbname):
@@ -353,44 +415,16 @@ def _db_is_locked(dbname):
     return locked
 
 
-# OSXPHOTOS_XATTR_UUID = "com.osxphotos.uuid"
-
-# def get_uuid_for_file(filepath):
-#     """ returns UUID associated with an exported file
-#         filepath: path to exported photo
-#     """
-#     attr = xattr.xattr(filepath)
-#     try:
-#         uuid_bytes = attr[OSXPHOTOS_XATTR_UUID]
-#         uuid_str = uuid_bytes.decode('utf-8')
-#     except KeyError:
-#         uuid_str = None
-#     return uuid_str
-
-# def set_uuid_for_file(filepath, uuid):
-#     """ sets the UUID associated with an exported file
-#         filepath: path to exported photo
-#         uuid: uuid string for photo
-#     """
-#     if not os.path.exists(filepath):
-#         raise FileNotFoundError(f"Missing file: {filepath}")
-
-#     attr = xattr.xattr(filepath)
-#     uuid_bytes = bytes(uuid, 'utf-8')
-#     attr.set(OSXPHOTOS_XATTR_UUID, uuid_bytes)
-
-
 def normalize_unicode(value):
     """normalize unicode data"""
-    if value is not None:
+    if value is None:
-        if isinstance(value, (tuple, list)):
-            return tuple(unicodedata.normalize(UNICODE_FORMAT, v) for v in value)
-        elif isinstance(value, str):
-            return unicodedata.normalize(UNICODE_FORMAT, value)
-        else:
-            return value
-    else:
         return None
+    if isinstance(value, (tuple, list)):
+        return tuple(unicodedata.normalize(UNICODE_FORMAT, v) for v in value)
+    elif isinstance(value, str):
+        return unicodedata.normalize(UNICODE_FORMAT, value)
+    else:
+        return value
 
 
 def increment_filename_with_count(
@@ -411,8 +445,8 @@ def increment_filename_with_count(
     Note: This obviously is subject to race condition so using with caution.
     """
     dest = filepath if isinstance(filepath, pathlib.Path) else pathlib.Path(filepath)
-    dest_files = findfiles(f"{dest.stem}*", str(dest.parent))
+    dest_files = list_directory(dest.parent, startswith=dest.stem)
-    dest_files = [pathlib.Path(f).stem.lower() for f in dest_files]
+    dest_files = [f.stem.lower() for f in dest_files]
     dest_new = f"{dest.stem} ({count})" if count else dest.stem
     dest_new = normalize_fs_path(dest_new)
 
@@ -478,3 +512,9 @@ def load_function(pyfile: str, function_name: str) -> Callable:
         sys.path = syspath
 
     return func
+
+
+def format_sec_to_hhmmss(sec: float) -> str:
+    """Format seconds to hh:mm:ss"""
+    delta = datetime.timedelta(seconds=sec)
+    return str(delta).split(".")[0]

tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>585B80BF-8D1F-55EF-A9E8-6CF4E5523959</string>
 	<key>pid</key>
-	<integer>1961</integer>
+	<integer>14817</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2021-09-14T04:40:41Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
 	<date>2021-07-20T05:48:08Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
 	<date>2021-07-20T05:47:59Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2021-09-14T04:40:43Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 </dict>
 </plist>

tests/conftest.py

@@ -40,7 +40,7 @@ else:
 
 @pytest.fixture(autouse=True)
 def reset_singletons():
-    """ Need to clean up any ExifTool singletons between tests """
+    """Need to clean up any ExifTool singletons between tests"""
     _ExifToolProc.instance = None
 
 
@@ -73,7 +73,7 @@ def pytest_collection_modifyitems(config, items):
 
 
 def copy_photos_library(photos_library=TEST_LIBRARY, delay=0):
-    """ copy the test library and open Photos, returns path to copied library """
+    """copy the test library and open Photos, returns path to copied library"""
     script = AppleScript(
         """
         tell application "Photos"
@@ -118,3 +118,9 @@ def copy_photos_library(photos_library=TEST_LIBRARY, delay=0):
 @pytest.fixture
 def addalbum_library():
     copy_photos_library(delay=10)
+
+
+def copy_photos_library_to_path(photos_library_path: str, dest_path: str) -> str:
+    """Copy a photos library to a folder"""
+    ditto(photos_library_path, dest_path)
+    return dest_path

tests/generate_sidecars_for_test.py

@@ -7,6 +7,7 @@
 import pathlib
 
 import osxphotos
+from osxphotos.photoexporter import PhotoExporter, ExportOptions
 
 PHOTOS_DB_15_7 = "./tests/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_DB_14_6 = "./tests/Test-10.14.6.photoslibrary/database/photos.db"
@@ -31,7 +32,7 @@ SIDECAR_DIR = "tests/sidecars"
 
 
 def generate_sidecars(dbname, uuid_dict):
-    """ generate XMP and JSON sidecars for testing """
+    """generate XMP and JSON sidecars for testing"""
     photosdb = osxphotos.PhotosDB(dbname)
 
     for _, uuid in uuid_dict.items():
@@ -39,7 +40,8 @@ def generate_sidecars(dbname, uuid_dict):
 
         # plain xmp
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}.xmp")
-        xmp = photo._xmp_sidecar()
+        exporter = PhotoExporter(photo)
+        xmp = exporter._xmp_sidecar()
         with open(sidecar, "w") as file:
             file.write(xmp)
 
@@ -47,63 +49,76 @@ def generate_sidecars(dbname, uuid_dict):
         ext = osxphotos.uti.get_preferred_uti_extension(photo.uti)
         ext = "jpg" if ext == "jpeg" else ext
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_ext.xmp")
-        xmp = photo._xmp_sidecar(extension=ext)
+        xmp = exporter._xmp_sidecar(extension=ext)
         with open(sidecar, "w") as file:
             file.write(xmp)
 
         # persons_as_keywords
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_persons_as_keywords.xmp")
-        xmp = photo._xmp_sidecar(use_persons_as_keywords=True, extension=ext)
+        xmp = exporter._xmp_sidecar(
+            ExportOptions(use_persons_as_keywords=True), extension=ext
+        )
         with open(sidecar, "w") as file:
             file.write(xmp)
 
         # albums_as_keywords
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_albums_as_keywords.xmp")
-        xmp = photo._xmp_sidecar(use_albums_as_keywords=True, extension=ext)
+        xmp = exporter._xmp_sidecar(
+            ExportOptions(use_albums_as_keywords=True), extension=ext
+        )
         with open(sidecar, "w") as file:
             file.write(xmp)
 
         # keyword_template
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_keyword_template.xmp")
-        xmp = photo._xmp_sidecar(
+        xmp = exporter._xmp_sidecar(
-            keyword_template=["{created.year}", "{folder_album}"], extension=ext
+            ExportOptions(keyword_template=["{created.year}", "{folder_album}"]),
+            extension=ext,
         )
         with open(sidecar, "w") as file:
             file.write(xmp)
 
         # generate JSON files
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}.json")
-        json_ = photo._exiftool_json_sidecar()
+        json_ = exporter._exiftool_json_sidecar()
         with open(sidecar, "w") as file:
             file.write(json_)
 
         # no tag groups
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_no_tag_groups.json")
-        json_ = photo._exiftool_json_sidecar(tag_groups=False)
+        json_ = exporter._exiftool_json_sidecar(tag_groups=False)
         with open(sidecar, "w") as file:
             file.write(json_)
 
         # ignore_date_modified
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_ignore_date_modified.json")
-        json_ = photo._exiftool_json_sidecar(ignore_date_modified=True)
+        json_ = exporter._exiftool_json_sidecar(
+            ExportOptions(ignore_date_modified=True)
+        )
         with open(sidecar, "w") as file:
             file.write(json_)
 
         # keyword_template
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_keyword_template.json")
-        json_ = photo._exiftool_json_sidecar(keyword_template=["{folder_album}"])
+        json_ = exporter._exiftool_json_sidecar(
+            ExportOptions(keyword_template=["{folder_album}"])
+        )
         with open(sidecar, "w") as file:
             file.write(json_)
 
         # persons_as_keywords
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_persons_as_keywords.json")
-        json_ = photo._exiftool_json_sidecar(use_persons_as_keywords=True)
+        json_ = exporter._exiftool_json_sidecar(
+            ExportOptions(use_persons_as_keywords=True)
+        )
         with open(sidecar, "w") as file:
             file.write(json_)
 
         # albums_as_keywords
         sidecar = str(pathlib.Path(SIDECAR_DIR) / f"{uuid}_albums_as_keywords.json")
-        json_ = photo._exiftool_json_sidecar(use_albums_as_keywords=True)
+        json_ = exporter._exiftool_json_sidecar(
+            ExportOptions(use_albums_as_keywords=True)
+        )
         with open(sidecar, "w") as file:
             file.write(json_)
 

tests/sidecars/15uNd7%8RguTEgNPKHfTWw.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Kids"], "XMP:Subject": ["Kids"], "XMP:TagsList": ["Kids"], "XMP:PersonInImage": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
+[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Kids"], "XMP:Subject": ["Kids"], "XMP:TagsList": ["Kids"], "XMP:PersonInImage": ["Katie"], "XMP:RegionAppliedToDimensionsW": 1365, "XMP:RegionAppliedToDimensionsH": 2048, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Katie"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.5832663178443909], "XMP:RegionAreaY": [0.27730926126241684], "XMP:RegionAreaW": [0.24365156888961792], "XMP:RegionAreaH": [0.16239472242887132], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw.xmp

@@ -52,6 +52,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>2048</stDim:h>
+    <stDim:w>1365</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_albums_as_keywords.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:Subject": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:TagsList": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:PersonInImage": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
+[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:Subject": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:TagsList": ["AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:PersonInImage": ["Katie"], "XMP:RegionAppliedToDimensionsW": 1365, "XMP:RegionAppliedToDimensionsH": 2048, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Katie"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.5832663178443909], "XMP:RegionAreaY": [0.27730926126241684], "XMP:RegionAreaW": [0.24365156888961792], "XMP:RegionAreaH": [0.16239472242887132], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_albums_as_keywords.xmp

@@ -58,6 +58,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>2048</stDim:h>
+    <stDim:w>1365</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_ext.xmp

@@ -52,6 +52,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>2048</stDim:h>
+    <stDim:w>1365</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_ignore_date_modified.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Kids"], "XMP:Subject": ["Kids"], "XMP:TagsList": ["Kids"], "XMP:PersonInImage": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
+[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Kids"], "XMP:Subject": ["Kids"], "XMP:TagsList": ["Kids"], "XMP:PersonInImage": ["Katie"], "XMP:RegionAppliedToDimensionsW": 1365, "XMP:RegionAppliedToDimensionsH": 2048, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Katie"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.5832663178443909], "XMP:RegionAreaY": [0.27730926126241684], "XMP:RegionAreaW": [0.24365156888961792], "XMP:RegionAreaH": [0.16239472242887132], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_keyword_template.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:Subject": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:TagsList": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:PersonInImage": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
+[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:Subject": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:TagsList": ["Folder1/SubFolder2/AlbumInFolder", "Kids", "Pumpkin Farm", "Test Album (1)"], "XMP:PersonInImage": ["Katie"], "XMP:RegionAppliedToDimensionsW": 1365, "XMP:RegionAppliedToDimensionsH": 2048, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Katie"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.5832663178443909], "XMP:RegionAreaY": [0.27730926126241684], "XMP:RegionAreaW": [0.24365156888961792], "XMP:RegionAreaH": [0.16239472242887132], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_keyword_template.xmp

@@ -60,6 +60,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>2048</stDim:h>
+    <stDim:w>1365</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_no_tag_groups.json

@@ -1 +1 @@
-[{"ImageDescription": "Girl holding pumpkin", "Description": "Girl holding pumpkin", "Caption-Abstract": "Girl holding pumpkin", "Title": "I found one!", "ObjectName": "I found one!", "Keywords": ["Kids"], "Subject": ["Kids"], "TagsList": ["Kids"], "PersonInImage": ["Katie"], "DateTimeOriginal": "2018:09:28 16:07:07", "CreateDate": "2018:09:28 16:07:07", "OffsetTimeOriginal": "-04:00", "DateCreated": "2018:09:28", "TimeCreated": "16:07:07-04:00", "ModifyDate": "2018:09:28 16:07:07"}]
+[{"ImageDescription": "Girl holding pumpkin", "Description": "Girl holding pumpkin", "Caption-Abstract": "Girl holding pumpkin", "Title": "I found one!", "ObjectName": "I found one!", "Keywords": ["Kids"], "Subject": ["Kids"], "TagsList": ["Kids"], "PersonInImage": ["Katie"], "RegionAppliedToDimensionsW": 1365, "RegionAppliedToDimensionsH": 2048, "RegionAppliedToDimensionsUnit": "pixel", "RegionName": ["Katie"], "RegionType": ["Face"], "RegionAreaX": [0.5832663178443909], "RegionAreaY": [0.27730926126241684], "RegionAreaW": [0.24365156888961792], "RegionAreaH": [0.16239472242887132], "RegionAreaUnit": ["normalized"], "RegionPersonDisplayName": ["Katie"], "DateTimeOriginal": "2018:09:28 16:07:07", "CreateDate": "2018:09:28 16:07:07", "OffsetTimeOriginal": "-04:00", "DateCreated": "2018:09:28", "TimeCreated": "16:07:07-04:00", "ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_persons_as_keywords.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Katie", "Kids"], "XMP:Subject": ["Katie", "Kids"], "XMP:TagsList": ["Katie", "Kids"], "XMP:PersonInImage": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]
+[{"EXIF:ImageDescription": "Girl holding pumpkin", "XMP:Description": "Girl holding pumpkin", "IPTC:Caption-Abstract": "Girl holding pumpkin", "XMP:Title": "I found one!", "IPTC:ObjectName": "I found one!", "IPTC:Keywords": ["Katie", "Kids"], "XMP:Subject": ["Katie", "Kids"], "XMP:TagsList": ["Katie", "Kids"], "XMP:PersonInImage": ["Katie"], "XMP:RegionAppliedToDimensionsW": 1365, "XMP:RegionAppliedToDimensionsH": 2048, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Katie"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.5832663178443909], "XMP:RegionAreaY": [0.27730926126241684], "XMP:RegionAreaW": [0.24365156888961792], "XMP:RegionAreaH": [0.16239472242887132], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Katie"], "EXIF:DateTimeOriginal": "2018:09:28 16:07:07", "EXIF:CreateDate": "2018:09:28 16:07:07", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2018:09:28", "IPTC:TimeCreated": "16:07:07-04:00", "EXIF:ModifyDate": "2018:09:28 16:07:07"}]

tests/sidecars/15uNd7%8RguTEgNPKHfTWw_persons_as_keywords.xmp

@@ -54,6 +54,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>2048</stDim:h>
+    <stDim:w>1365</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]
+[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "XMP:RegionAppliedToDimensionsW": 1526, "XMP:RegionAppliedToDimensionsH": 1325, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Maria"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.40229974687099457], "XMP:RegionAreaY": [0.41379398107528687], "XMP:RegionAreaW": [0.3420099108278517], "XMP:RegionAreaH": [0.39389216899871826], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ.xmp

@@ -52,6 +52,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>1325</stDim:h>
+    <stDim:w>1526</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_albums_as_keywords.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]
+[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "XMP:RegionAppliedToDimensionsW": 1526, "XMP:RegionAppliedToDimensionsH": 1325, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Maria"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.40229974687099457], "XMP:RegionAreaY": [0.41379398107528687], "XMP:RegionAreaW": [0.3420099108278517], "XMP:RegionAreaH": [0.39389216899871826], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_albums_as_keywords.xmp

@@ -52,6 +52,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>1325</stDim:h>
+    <stDim:w>1526</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_ext.xmp

@@ -52,6 +52,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>1325</stDim:h>
+    <stDim:w>1526</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_ignore_date_modified.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:04:15 14:40:24"}]
+[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "XMP:RegionAppliedToDimensionsW": 1526, "XMP:RegionAppliedToDimensionsH": 1325, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Maria"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.40229974687099457], "XMP:RegionAreaY": [0.41379398107528687], "XMP:RegionAreaW": [0.3420099108278517], "XMP:RegionAreaH": [0.39389216899871826], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:04:15 14:40:24"}]

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_keyword_template.json

@@ -1 +1 @@
-[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]
+[{"EXIF:ImageDescription": "Bride Wedding day", "XMP:Description": "Bride Wedding day", "IPTC:Caption-Abstract": "Bride Wedding day", "IPTC:Keywords": ["wedding"], "XMP:Subject": ["wedding"], "XMP:TagsList": ["wedding"], "XMP:PersonInImage": ["Maria"], "XMP:RegionAppliedToDimensionsW": 1526, "XMP:RegionAppliedToDimensionsH": 1325, "XMP:RegionAppliedToDimensionsUnit": "pixel", "XMP:RegionName": ["Maria"], "XMP:RegionType": ["Face"], "XMP:RegionAreaX": [0.40229974687099457], "XMP:RegionAreaY": [0.41379398107528687], "XMP:RegionAreaW": [0.3420099108278517], "XMP:RegionAreaH": [0.39389216899871826], "XMP:RegionAreaUnit": ["normalized"], "XMP:RegionPersonDisplayName": ["Maria"], "EXIF:DateTimeOriginal": "2019:04:15 14:40:24", "EXIF:CreateDate": "2019:04:15 14:40:24", "EXIF:OffsetTimeOriginal": "-04:00", "IPTC:DateCreated": "2019:04:15", "IPTC:TimeCreated": "14:40:24-04:00", "EXIF:ModifyDate": "2019:11:27 01:30:16"}]

tests/sidecars/6bxcNnzRQKGnK4uPrCJ9UQ_keyword_template.xmp

@@ -54,6 +54,8 @@
  xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
     <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:AppliedToDimensions rdf:parseType="Resource">
+    <stDim:h>1325</stDim:h>
+    <stDim:w>1526</stDim:w>
     <stDim:unit>pixel</stDim:unit>
     </mwg-rs:AppliedToDimensions>
     <mwg-rs:RegionList>