Added --force-update, #621

Updated CHANGELOG.md [skip ci]
2022-02-12 21:01:16 -08:00 · 2022-02-12 17:59:48 -08:00 · 2022-02-12 17:49:40 -08:00 · 2022-02-12 00:23:50 -08:00 · 2022-02-11 23:10:09 -08:00 · 2022-02-11 06:18:17 -08:00
78 changed files with 6681 additions and 955 deletions
--- a/.all-contributorsrc
+++ b/.all-contributorsrc
@@ -257,7 +257,9 @@
      "avatar_url": "https://avatars.githubusercontent.com/u/21261491?v=4",
      "profile": "https://github.com/oPromessa",
      "contributions": [
-        "bug"
+        "bug",
+        "ideas",
+        "test"
      ]
    },
    {
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,62 @@ All notable changes to this project will be documented in this file. Dates are d

 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).

+#### [v0.45.9](https://github.com/RhetTbull/osxphotos/compare/v0.45.8...v0.45.9)
+
+> 12 February 2022
+
+- Added --force-update, #621 [`bfa888a`](https://github.com/RhetTbull/osxphotos/commit/bfa888adc5658a2845dcaa9b7ea360926ed4f000)
+- Refactored fix for #627 [`5fb686a`](https://github.com/RhetTbull/osxphotos/commit/5fb686ac0c231932c2695fc550a0824307bd3c5f)
+- Fix for #630 [`ac4083b`](https://github.com/RhetTbull/osxphotos/commit/ac4083bfbbabc8550718f0f7f8aadc635c05eb25)
+
+#### [v0.45.8](https://github.com/RhetTbull/osxphotos/compare/v0.45.6...v0.45.8)
+
+> 5 February 2022
+
+- Fixed exiftool to ignore unsupported file types, #615 [`1ae6270`](https://github.com/RhetTbull/osxphotos/commit/1ae627056113fc4655f1b24cfbbdf0efc04489e7)
+- Updated tests [`55a601c`](https://github.com/RhetTbull/osxphotos/commit/55a601c07ea1384623c55d5c1d26b568df5d7823)
+- Additional fix for #615 [`1d6bc4e`](https://github.com/RhetTbull/osxphotos/commit/1d6bc4e09e3c2359a21f842fadd781920606812e)
+
+#### [v0.45.6](https://github.com/RhetTbull/osxphotos/compare/v0.45.5...v0.45.6)
+
+> 5 February 2022
+
+- Fix for unicode in query strings, #618 [`9b247ac`](https://github.com/RhetTbull/osxphotos/commit/9b247acd1cc4b2def59fdd18a6fb3c8eb9914f11)
+- Fix for --name searching only original_filename on Photos 5+, #594 [`cd02144`](https://github.com/RhetTbull/osxphotos/commit/cd02144ac33cc1c13a20358133971c84d35b8a57)
+
+#### [v0.45.5](https://github.com/RhetTbull/osxphotos/compare/v0.45.4...v0.45.5)
+
+> 5 February 2022
+
+- Fix for #561, no really, I mean it this time [`b3d3e14`](https://github.com/RhetTbull/osxphotos/commit/b3d3e14ffe41fbb22edb614b24f3985f379766a2)
+- Updated docs [skip ci] [`2b9ea11`](https://github.com/RhetTbull/osxphotos/commit/2b9ea11701799af9a661a8e2af70fca97235f487)
+- Updated tests for #561 [skip ci] [`77a49a0`](https://github.com/RhetTbull/osxphotos/commit/77a49a09a1bee74113a7114c543fbc25fa410ffc)
+
+#### [v0.45.4](https://github.com/RhetTbull/osxphotos/compare/v0.45.3...v0.45.4)
+
+> 3 February 2022
+
+- docs: add oPromessa as a contributor for ideas, test [`#611`](https://github.com/RhetTbull/osxphotos/pull/611)
+- Fix for filenames with special characters, #561, #618 [`f3063d3`](https://github.com/RhetTbull/osxphotos/commit/f3063d35be3c96342d83dbd87ddd614a2001bff4)
+- Updated docs [skip ci] [`06c5bbf`](https://github.com/RhetTbull/osxphotos/commit/06c5bbfcfdf591a4a5d43f1456adaa27385fe01a)
+- Added progress counter, #601 [`7ab5007`](https://github.com/RhetTbull/osxphotos/commit/7ab500740b28594dcd778140e10991f839220e9d)
+- Updated known issues [skip ci] [`e32090b`](https://github.com/RhetTbull/osxphotos/commit/e32090bf39cb786171b49443f878ffdbab774420)
+
+#### [v0.45.3](https://github.com/RhetTbull/osxphotos/compare/v0.45.2...v0.45.3)
+
+> 29 January 2022
+
+- Added --timestamp option for --verbose, #600 [`d8c2f99`](https://github.com/RhetTbull/osxphotos/commit/d8c2f99c06bc6f72bf2cb1a13c5765824fe3cbba)
+- Updated docs [skip ci] [`5fc2813`](https://github.com/RhetTbull/osxphotos/commit/5fc28139ea0374bc3e228c0432b8a41ada430389)
+- Updated formatting for elapsed time, #604 [`16d3f74`](https://github.com/RhetTbull/osxphotos/commit/16d3f743664396d43b3b3028a5e7a919ec56d9e1)
+
+#### [v0.45.2](https://github.com/RhetTbull/osxphotos/compare/v0.45.0...v0.45.2)
+
+> 29 January 2022
+
+- Implemented #605, refactor out export2 [`235dea3`](https://github.com/RhetTbull/osxphotos/commit/235dea329c98ab8fa61565c09a1b4a83e5d99043)
+- Fix for #564, --preview with --download-missing [`5afdf6f`](https://github.com/RhetTbull/osxphotos/commit/5afdf6fc20a3cb6eb2b0217d8b3be20295eb7ba4)
+
 #### [v0.45.0](https://github.com/RhetTbull/osxphotos/compare/v0.44.13...v0.45.0)

 > 28 January 2022
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,7 +1,7 @@
-include README.md
-include README.rst
-include osxphotos/templates/*
+include osxphotos/*.json
+include osxphotos/*.md
 include osxphotos/phototemplate.tx
-include osxphotos/phototemplate.md
-include osxphotos/tutorial.md
-include osxphotos/queries/*
+include osxphotos/queries/*
+include osxphotos/templates/*
+include README.md
+include README.rst
--- a/README.md
+++ b/README.md
@@ -601,6 +601,7 @@ Options:
                                  library, 2. system library, 3.
                                  ~/Pictures/Photos Library.photoslibrary
  -V, --verbose                   Print verbose output.
+  --timestamp                     Add time stamp to verbose output
  --keyword KEYWORD               Search for photos with keyword KEYWORD. If
                                  more than one keyword, treated as "OR", e.g.
                                  find photos matching any keyword
@@ -782,8 +783,15 @@ Options:
                                  folder.
  --deleted-only                  Include only photos from the 'Recently
                                  Deleted' folder.
-  --update                        Only export new or updated files. See notes
-                                  below on export and --update.
+  --update                        Only export new or updated files. See also
+                                  --force-update and notes below on export and
+                                  --update.
+  --force-update                  Only export new or updated files. Unlike
+                                  --update, --force-update will re-export photos
+                                  if their metadata has changed even if this
+                                  would not otherwise trigger an export. See
+                                  also --update and notes below on export and
+                                  --update.
  --ignore-signature              When used with '--update', ignores file
                                  signature when updating files. This is useful
                                  if you have processed or edited exported
@@ -1724,7 +1732,7 @@ Substitution                    Description
 {lf}                            A line feed: '\n', alias for {newline}
 {cr}                            A carriage return: '\r'
 {crlf}                          a carriage return + line feed: '\r\n'
-{osxphotos_version}             The osxphotos version, e.g. '0.45.0'
+{osxphotos_version}             The osxphotos version, e.g. '0.45.10'
 {osxphotos_cmd_line}            The full command line used to run osxphotos

 The following substitutions may result in multiple values. Thus if specified for
@@ -3628,7 +3636,7 @@ The following template field substitutions are availabe for use the templating s
 |{lf}|A line feed: '\n', alias for {newline}|
 |{cr}|A carriage return: '\r'|
 |{crlf}|a carriage return + line feed: '\r\n'|
-|{osxphotos_version}|The osxphotos version, e.g. '0.45.0'|
+|{osxphotos_version}|The osxphotos version, e.g. '0.45.10'|
 |{osxphotos_cmd_line}|The full command line used to run osxphotos|
 |{album}|Album(s) photo is contained in|
 |{folder_album}|Folder path + album photo is contained in. e.g. 'Folder/Subfolder/Album' or just 'Album' if no enclosing folder|
@@ -3947,7 +3955,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
    <td align="center"><a href="https://github.com/mkirkland4874"><img src="https://avatars.githubusercontent.com/u/36466711?v=4?s=75" width="75px;" alt=""/><br /><sub><b>mkirkland4874</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Amkirkland4874" title="Bug reports">🐛</a> <a href="#example-mkirkland4874" title="Examples">💡</a></td>
    <td align="center"><a href="https://github.com/jcommisso07"><img src="https://avatars.githubusercontent.com/u/3111054?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Joseph Commisso</b></sub></a><br /><a href="#data-jcommisso07" title="Data">🔣</a></td>
    <td align="center"><a href="https://github.com/dssinger"><img src="https://avatars.githubusercontent.com/u/1817903?v=4?s=75" width="75px;" alt=""/><br /><sub><b>David Singer</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Adssinger" title="Bug reports">🐛</a></td>
-    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a></td>
+    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a> <a href="#ideas-oPromessa" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/RhetTbull/osxphotos/commits?author=oPromessa" title="Tests">⚠️</a></td>
    <td align="center"><a href="http://spencerchang.me"><img src="https://avatars.githubusercontent.com/u/14796580?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Spencer Chang</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Aspencerc99" title="Bug reports">🐛</a></td>
  </tr>
  <tr>
@@ -3973,7 +3981,6 @@ My goal is make osxphotos as reliable and comprehensive as possible.  The test s

 - Audio-only files are not handled.  It is possible to store audio-only files in Photos.  osxphotos currently only handles images and videos. See [Issue #436](https://github.com/RhetTbull/osxphotos/issues/436)
 - Face coordinates (mouth, left eye, right eye) may not be correct for images where the head is tilted.  See [Issue #196](https://github.com/RhetTbull/osxphotos/issues/196).
- Raw images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the raw image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the raw image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75).

 ## Implementation Notes
--- a/docs/.buildinfo
+++ b/docs/.buildinfo
@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: fb08da4139dc0e6a5dd26bd81551df02
+config: 5bcb39b8d7265fb2c5780a64f97662ef
 tags: 645f666f9bcd5a90fca523b33c5a78b7
--- a/docs/_static/documentation_options.js
+++ b/docs/_static/documentation_options.js
@@ -1,6 +1,6 @@
 var DOCUMENTATION_OPTIONS = {
    URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
-    VERSION: '0.45.0',
+    VERSION: '0.45.10',
    LANGUAGE: 'None',
    COLLAPSE_INDEX: false,
    BUILDER: 'html',
--- a/docs/cli.html
+++ b/docs/cli.html
@@ -6,7 +6,7 @@
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

-    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.45.0 documentation</title>
+    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
--- a/docs/genindex.html
+++ b/docs/genindex.html
@@ -5,7 +5,7 @@
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Index &#8212; osxphotos 0.45.0 documentation</title>
+    <title>Index &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
--- a/docs/index.html
+++ b/docs/index.html
@@ -6,7 +6,7 @@
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

-    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.45.0 documentation</title>
+    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
--- a/docs/modules.html
+++ b/docs/modules.html
@@ -6,7 +6,7 @@
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

-    <title>osxphotos &#8212; osxphotos 0.45.0 documentation</title>
+    <title>osxphotos &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
--- a/docs/reference.html
+++ b/docs/reference.html
@@ -6,7 +6,7 @@
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

-    <title>osxphotos package &#8212; osxphotos 0.45.0 documentation</title>
+    <title>osxphotos package &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
--- a/docs/search.html
+++ b/docs/search.html
@@ -5,7 +5,7 @@
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Search &#8212; osxphotos 0.45.0 documentation</title>
+    <title>Search &#8212; osxphotos 0.45.10 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
    
--- a/osxphotos.spec
+++ b/osxphotos.spec
@@ -14,6 +14,7 @@ datas = [
    ("osxphotos/phototemplate.tx", "osxphotos"),
    ("osxphotos/phototemplate.md", "osxphotos"),
    ("osxphotos/tutorial.md", "osxphotos"),
+    ("osxphotos/exiftool_filetypes.json", "osxphotos"),
 ]
 package_imports = [["photoscript", ["photoscript.applescript"]]]
 for package, files in package_imports:
--- a/osxphotos/_version.py
+++ b/osxphotos/_version.py
@@ -1,3 +1,3 @@
 """ version info """

-__version__ = "0.45.2"
+__version__ = "0.45.10"
--- a/osxphotos/cli.py
+++ b/osxphotos/cli.py
@@ -79,6 +79,7 @@ from .sqlgrep import sqlgrep
 from .uti import get_preferred_uti_extension
 from .utils import (
    expand_and_validate_filepath,
+    format_sec_to_hhmmss,
    load_function,
    normalize_fs_path,
 )
@@ -132,6 +133,7 @@ __all__ = [
 # global variable to control verbose output
 # set via --verbose/-V
 VERBOSE = False
+VERBOSE_TIMESTAMP = False

 # used to show/hide hidden commands
 OSXPHOTOS_HIDDEN = not bool(os.getenv("OSXPHOTOS_SHOW_HIDDEN", default=False))
@@ -146,8 +148,10 @@ def verbose_(*args, **kwargs):
    """print output if verbose flag set"""
    if VERBOSE:
        styled_args = []
+        timestamp = str(datetime.datetime.now()) + " -- " if VERBOSE_TIMESTAMP else ""
        for arg in args:
            if type(arg) == str:
+                arg = timestamp + arg
                if "error" in arg.lower():
                    arg = click.style(arg, fg=CLI_COLOR_ERROR)
                elif "warning" in arg.lower():
@@ -676,6 +680,7 @@ def cli(ctx, db, json_, debug):
@cli.command(cls=ExportCommand)
@DB_OPTION
@click.option("--verbose", "-V", "verbose", is_flag=True, help="Print verbose output.")
+@click.option("--timestamp", is_flag=True, help="Add time stamp to verbose output")
@QUERY_OPTIONS
@click.option(
    "--missing",
@@ -686,7 +691,15 @@ def cli(ctx, db, json_, debug):
@click.option(
    "--update",
    is_flag=True,
-    help="Only export new or updated files. See notes below on export and --update.",
+    help="Only export new or updated files. "
+    "See also --force-update and notes below on export and --update.",
+)
+@click.option(
+    "--force-update",
+    is_flag=True,
+    help="Only export new or updated files. Unlike --update, --force-update will re-export photos "
+    "if their metadata has changed even if this would not otherwise trigger an export. "
+    "See also --update and notes below on export and --update.",
 )
@click.option(
    "--ignore-signature",
@@ -1227,8 +1240,10 @@ def export(
    from_time,
    to_time,
    verbose,
+    timestamp,
    missing,
    update,
+    force_update,
    ignore_signature,
    only_new,
    dry_run,
@@ -1373,7 +1388,9 @@ def export(
    )

    global VERBOSE
+    global VERBOSE_TIMESTAMP
    VERBOSE = bool(verbose)
+    VERBOSE_TIMESTAMP = timestamp

    if load_config:
        try:
@@ -1390,133 +1407,134 @@ def export(

        # re-set the local vars to the corresponding config value
        # this isn't elegant but avoids having to rewrite this function to use cfg.varname for every parameter
-        db = cfg.db
-        photos_library = cfg.photos_library
-        keyword = cfg.keyword
-        person = cfg.person
+        add_exported_to_album = cfg.add_exported_to_album
+        add_missing_to_album = cfg.add_missing_to_album
+        add_skipped_to_album = cfg.add_skipped_to_album
        album = cfg.album
-        folder = cfg.folder
-        name = cfg.name
-        uuid = cfg.uuid
-        uuid_from_file = cfg.uuid_from_file
-        title = cfg.title
-        no_title = cfg.no_title
-        description = cfg.description
-        no_description = cfg.no_description
-        uti = cfg.uti
-        ignore_case = cfg.ignore_case
-        edited = cfg.edited
-        external_edit = cfg.external_edit
-        favorite = cfg.favorite
-        not_favorite = cfg.not_favorite
-        hidden = cfg.hidden
-        not_hidden = cfg.not_hidden
-        shared = cfg.shared
-        not_shared = cfg.not_shared
-        from_date = cfg.from_date
-        to_date = cfg.to_date
-        from_time = cfg.from_time
-        to_time = cfg.to_time
-        verbose = cfg.verbose
-        missing = cfg.missing
-        update = cfg.update
-        ignore_signature = cfg.ignore_signature
-        dry_run = cfg.dry_run
-        export_as_hardlink = cfg.export_as_hardlink
-        touch_file = cfg.touch_file
-        overwrite = cfg.overwrite
-        retry = cfg.retry
-        export_by_date = cfg.export_by_date
-        skip_edited = cfg.skip_edited
-        skip_original_if_edited = cfg.skip_original_if_edited
-        skip_bursts = cfg.skip_bursts
-        skip_live = cfg.skip_live
-        skip_raw = cfg.skip_raw
-        skip_uuid = cfg.skip_uuid
-        skip_uuid_from_file = cfg.skip_uuid_from_file
-        person_keyword = cfg.person_keyword
        album_keyword = cfg.album_keyword
-        keyword_template = cfg.keyword_template
-        replace_keywords = cfg.replace_keywords
-        description_template = cfg.description_template
-        finder_tag_template = cfg.finder_tag_template
-        finder_tag_keywords = cfg.finder_tag_keywords
-        xattr_template = cfg.xattr_template
-        current_name = cfg.current_name
-        convert_to_jpeg = cfg.convert_to_jpeg
-        jpeg_quality = cfg.jpeg_quality
-        sidecar = cfg.sidecar
-        sidecar_drop_ext = cfg.sidecar_drop_ext
-        only_photos = cfg.only_photos
-        only_movies = cfg.only_movies
+        beta = cfg.beta
        burst = cfg.burst
-        not_burst = cfg.not_burst
-        live = cfg.live
-        not_live = cfg.not_live
-        download_missing = cfg.download_missing
-        exiftool = cfg.exiftool
-        exiftool_path = cfg.exiftool_path
-        exiftool_option = cfg.exiftool_option
-        exiftool_merge_keywords = cfg.exiftool_merge_keywords
-        exiftool_merge_persons = cfg.exiftool_merge_persons
-        ignore_date_modified = cfg.ignore_date_modified
-        portrait = cfg.portrait
-        not_portrait = cfg.not_portrait
-        screenshot = cfg.screenshot
-        not_screenshot = cfg.not_screenshot
-        slow_mo = cfg.slow_mo
-        not_slow_mo = cfg.not_slow_mo
-        time_lapse = cfg.time_lapse
-        not_time_lapse = cfg.not_time_lapse
-        hdr = cfg.hdr
-        not_hdr = cfg.not_hdr
-        selfie = cfg.selfie
-        not_selfie = cfg.not_selfie
-        panorama = cfg.panorama
-        not_panorama = cfg.not_panorama
-        has_raw = cfg.has_raw
-        directory = cfg.directory
-        filename_template = cfg.filename_template
-        jpeg_ext = cfg.jpeg_ext
-        strip = cfg.strip
-        edited_suffix = cfg.edited_suffix
-        original_suffix = cfg.original_suffix
-        place = cfg.place
-        no_place = cfg.no_place
-        location = cfg.location
-        no_location = cfg.no_location
-        has_comment = cfg.has_comment
-        no_comment = cfg.no_comment
-        has_likes = cfg.has_likes
-        no_likes = cfg.no_likes
-        label = cfg.label
+        cleanup = cfg.cleanup
+        convert_to_jpeg = cfg.convert_to_jpeg
+        current_name = cfg.current_name
+        db = cfg.db
        deleted = cfg.deleted
        deleted_only = cfg.deleted_only
-        use_photos_export = cfg.use_photos_export
-        use_photokit = cfg.use_photokit
-        report = cfg.report
-        cleanup = cfg.cleanup
-        add_exported_to_album = cfg.add_exported_to_album
-        add_skipped_to_album = cfg.add_skipped_to_album
-        add_missing_to_album = cfg.add_missing_to_album
-        exportdb = cfg.exportdb
-        beta = cfg.beta
-        only_new = cfg.only_new
-        in_album = cfg.in_album
-        not_in_album = cfg.not_in_album
-        min_size = cfg.min_size
-        max_size = cfg.max_size
-        regex = cfg.regex
-        selected = cfg.selected
-        exif = cfg.exif
-        query_eval = cfg.query_eval
-        query_function = cfg.query_function
+        description = cfg.description
+        description_template = cfg.description_template
+        directory = cfg.directory
+        download_missing = cfg.download_missing
+        dry_run = cfg.dry_run
        duplicate = cfg.duplicate
+        edited = cfg.edited
+        edited_suffix = cfg.edited_suffix
+        exif = cfg.exif
+        exiftool = cfg.exiftool
+        exiftool_merge_keywords = cfg.exiftool_merge_keywords
+        exiftool_merge_persons = cfg.exiftool_merge_persons
+        exiftool_option = cfg.exiftool_option
+        exiftool_path = cfg.exiftool_path
+        export_as_hardlink = cfg.export_as_hardlink
+        export_by_date = cfg.export_by_date
+        exportdb = cfg.exportdb
+        external_edit = cfg.external_edit
+        favorite = cfg.favorite
+        filename_template = cfg.filename_template
+        finder_tag_keywords = cfg.finder_tag_keywords
+        finder_tag_template = cfg.finder_tag_template
+        folder = cfg.folder
+        force_update = cfg.force_update
+        from_date = cfg.from_date
+        from_time = cfg.from_time
+        has_comment = cfg.has_comment
+        has_likes = cfg.has_likes
+        has_raw = cfg.has_raw
+        hdr = cfg.hdr
+        hidden = cfg.hidden
+        ignore_case = cfg.ignore_case
+        ignore_date_modified = cfg.ignore_date_modified
+        ignore_signature = cfg.ignore_signature
+        in_album = cfg.in_album
+        jpeg_ext = cfg.jpeg_ext
+        jpeg_quality = cfg.jpeg_quality
+        keyword = cfg.keyword
+        keyword_template = cfg.keyword_template
+        label = cfg.label
+        live = cfg.live
+        location = cfg.location
+        max_size = cfg.max_size
+        min_size = cfg.min_size
+        missing = cfg.missing
+        name = cfg.name
+        no_comment = cfg.no_comment
+        no_description = cfg.no_description
+        no_likes = cfg.no_likes
+        no_location = cfg.no_location
+        no_place = cfg.no_place
+        no_title = cfg.no_title
+        not_burst = cfg.not_burst
+        not_favorite = cfg.not_favorite
+        not_hdr = cfg.not_hdr
+        not_hidden = cfg.not_hidden
+        not_in_album = cfg.not_in_album
+        not_live = cfg.not_live
+        not_panorama = cfg.not_panorama
+        not_portrait = cfg.not_portrait
+        not_screenshot = cfg.not_screenshot
+        not_selfie = cfg.not_selfie
+        not_shared = cfg.not_shared
+        not_slow_mo = cfg.not_slow_mo
+        not_time_lapse = cfg.not_time_lapse
+        only_movies = cfg.only_movies
+        only_new = cfg.only_new
+        only_photos = cfg.only_photos
+        original_suffix = cfg.original_suffix
+        overwrite = cfg.overwrite
+        panorama = cfg.panorama
+        person = cfg.person
+        person_keyword = cfg.person_keyword
+        photos_library = cfg.photos_library
+        place = cfg.place
+        portrait = cfg.portrait
        post_command = cfg.post_command
        post_function = cfg.post_function
        preview = cfg.preview
-        preview_suffix = cfg.preview_suffix
        preview_if_missing = cfg.preview_if_missing
+        preview_suffix = cfg.preview_suffix
+        query_eval = cfg.query_eval
+        query_function = cfg.query_function
+        regex = cfg.regex
+        replace_keywords = cfg.replace_keywords
+        report = cfg.report
+        retry = cfg.retry
+        screenshot = cfg.screenshot
+        selected = cfg.selected
+        selfie = cfg.selfie
+        shared = cfg.shared
+        sidecar = cfg.sidecar
+        sidecar_drop_ext = cfg.sidecar_drop_ext
+        skip_bursts = cfg.skip_bursts
+        skip_edited = cfg.skip_edited
+        skip_live = cfg.skip_live
+        skip_original_if_edited = cfg.skip_original_if_edited
+        skip_raw = cfg.skip_raw
+        skip_uuid = cfg.skip_uuid
+        skip_uuid_from_file = cfg.skip_uuid_from_file
+        slow_mo = cfg.slow_mo
+        strip = cfg.strip
+        time_lapse = cfg.time_lapse
+        title = cfg.title
+        to_date = cfg.to_date
+        to_time = cfg.to_time
+        touch_file = cfg.touch_file
+        update = cfg.update
+        use_photokit = cfg.use_photokit
+        use_photos_export = cfg.use_photos_export
+        uti = cfg.uti
+        uuid = cfg.uuid
+        uuid_from_file = cfg.uuid_from_file
+        verbose = cfg.verbose
+        xattr_template = cfg.xattr_template

        # config file might have changed verbose
        VERBOSE = bool(verbose)
@@ -1556,8 +1574,8 @@ def export(
    dependent_options = [
        ("missing", ("download_missing", "use_photos_export")),
        ("jpeg_quality", ("convert_to_jpeg")),
-        ("ignore_signature", ("update")),
-        ("only_new", ("update")),
+        ("ignore_signature", ("update", "force_update")),
+        ("only_new", ("update", "force_update")),
        ("exiftool_option", ("exiftool")),
        ("exiftool_merge_keywords", ("exiftool", "sidecar")),
        ("exiftool_merge_persons", ("exiftool", "sidecar")),
@@ -1888,56 +1906,61 @@ def export(
            else None
        )

+        photo_num = 0
        # send progress bar output to /dev/null if verbose to hide the progress bar
        fp = open(os.devnull, "w") if verbose else None
-        with click.progressbar(photos, file=fp) as bar:
+        with click.progressbar(photos, show_pos=True, file=fp) as bar:
            for p in bar:
+                photo_num += 1
                export_results = export_photo(
                    photo=p,
                    dest=dest,
-                    verbose=verbose,
-                    export_by_date=export_by_date,
-                    sidecar=sidecar,
-                    sidecar_drop_ext=sidecar_drop_ext,
-                    update=update,
-                    ignore_signature=ignore_signature,
-                    export_as_hardlink=export_as_hardlink,
-                    overwrite=overwrite,
-                    export_edited=export_edited,
-                    skip_original_if_edited=skip_original_if_edited,
-                    original_name=original_name,
-                    export_live=export_live,
+                    album_keyword=album_keyword,
+                    convert_to_jpeg=convert_to_jpeg,
+                    description_template=description_template,
+                    directory=directory,
                    download_missing=download_missing,
-                    exiftool=exiftool,
+                    dry_run=dry_run,
+                    edited_suffix=edited_suffix,
                    exiftool_merge_keywords=exiftool_merge_keywords,
                    exiftool_merge_persons=exiftool_merge_persons,
-                    directory=directory,
-                    filename_template=filename_template,
-                    export_raw=export_raw,
-                    album_keyword=album_keyword,
-                    person_keyword=person_keyword,
-                    keyword_template=keyword_template,
-                    description_template=description_template,
-                    export_db=export_db,
-                    fileutil=fileutil,
-                    dry_run=dry_run,
-                    touch_file=touch_file,
-                    edited_suffix=edited_suffix,
-                    original_suffix=original_suffix,
-                    use_photos_export=use_photos_export,
-                    convert_to_jpeg=convert_to_jpeg,
-                    jpeg_quality=jpeg_quality,
-                    ignore_date_modified=ignore_date_modified,
-                    use_photokit=use_photokit,
                    exiftool_option=exiftool_option,
-                    strip=strip,
+                    exiftool=exiftool,
+                    export_as_hardlink=export_as_hardlink,
+                    export_by_date=export_by_date,
+                    export_db=export_db,
+                    export_dir=dest,
+                    export_edited=export_edited,
+                    export_live=export_live,
+                    export_preview=preview,
+                    export_raw=export_raw,
+                    filename_template=filename_template,
+                    fileutil=fileutil,
+                    force_update=force_update,
+                    ignore_date_modified=ignore_date_modified,
+                    ignore_signature=ignore_signature,
                    jpeg_ext=jpeg_ext,
+                    jpeg_quality=jpeg_quality,
+                    keyword_template=keyword_template,
+                    num_photos=num_photos,
+                    original_name=original_name,
+                    original_suffix=original_suffix,
+                    overwrite=overwrite,
+                    person_keyword=person_keyword,
+                    photo_num=photo_num,
+                    preview_if_missing=preview_if_missing,
+                    preview_suffix=preview_suffix,
                    replace_keywords=replace_keywords,
                    retry=retry,
-                    export_dir=dest,
-                    export_preview=preview,
-                    preview_suffix=preview_suffix,
-                    preview_if_missing=preview_if_missing,
+                    sidecar_drop_ext=sidecar_drop_ext,
+                    sidecar=sidecar,
+                    skip_original_if_edited=skip_original_if_edited,
+                    strip=strip,
+                    touch_file=touch_file,
+                    update=update,
+                    use_photokit=use_photokit,
+                    use_photos_export=use_photos_export,
+                    verbose=verbose,
                )

                if post_function:
@@ -2050,7 +2073,7 @@ def export(
            fp.close()

        photo_str_total = "photos" if len(photos) != 1 else "photo"
-        if update:
+        if update or force_update:
            summary = (
                f"Processed: {len(photos)} {photo_str_total}, "
                f"exported: {len(results.new)}, "
@@ -2069,12 +2092,14 @@ def export(
            summary += f", touched date: {len(results.touched)}"
        click.echo(summary)
        stop_time = time.perf_counter()
-        click.echo(f"Elapsed time: {(stop_time-start_time):.3f} seconds")
+        click.echo(f"Elapsed time: {format_sec_to_hhmmss(stop_time-start_time)}")
    else:
        click.echo("Did not find any photos to export")

    # cleanup files and do report if needed
    if cleanup:
+        db_file = str(pathlib.Path(export_db_path).resolve())
+        db_files = [db_file, db_file + "-wal", db_file + "-shm"]
        all_files = (
            results.exported
            + results.skipped
@@ -2093,7 +2118,7 @@ def export(
            + results.missing
            # include files that have error in case they exist from previous export
            + [r[0] for r in results.error]
-            + [str(pathlib.Path(export_db_path).resolve())]
+            + db_files
        )
        click.echo(f"Cleaning up {dest}")
        cleaned_files, cleaned_dirs = cleanup_files(dest, all_files, fileutil)
@@ -2578,6 +2603,7 @@ def export_photo(
    sidecar=None,
    sidecar_drop_ext=False,
    update=None,
+    force_update=None,
    ignore_signature=None,
    export_as_hardlink=None,
    overwrite=None,
@@ -2616,51 +2642,55 @@ def export_photo(
    export_preview=False,
    preview_suffix=None,
    preview_if_missing=False,
+    photo_num=1,
+    num_photos=1,
 ):
    """Helper function for export that does the actual export

    Args:
        photo: PhotoInfo object
        dest: destination path as string
-        verbose: boolean; print verbose output
-        export_by_date: boolean; create export folder in form dest/YYYY/MM/DD
-        sidecar: list zero, 1 or 2 of ["json","xmp"] of sidecar variety to export
-        sidecar_drop_ext: boolean; if True, drops photo extension from sidecar name
-        export_as_hardlink: boolean; hardlink files instead of copying them
-        overwrite: boolean; overwrite dest file if it already exists
-        original_name: boolean; use original filename instead of current filename
-        export_live: boolean; also export live video component if photo is a live photo
-                     live video will have same name as photo but with .mov extension
-        download_missing: attempt download of missing iCloud photos
-        exiftool: use exiftool to write EXIF metadata directly to exported photo
+        album_keyword: bool; if True, exports album names as keywords in metadata
+        convert_to_jpeg: bool; if True, converts non-jpeg images to jpeg
+        description_template: str; optional template string that will be rendered for use as photo description
        directory: template used to determine output directory
-        filename_template: template use to determine output file
-        export_raw: boolean; if True exports raw image associate with the photo
-        export_edited: boolean; if True exports edited version of photo if there is one
-        skip_original_if_edited: boolean; if True does not export original if photo has been edited
-        album_keyword: boolean; if True, exports album names as keywords in metadata
-        person_keyword: boolean; if True, exports person names as keywords in metadata
-        keyword_template: list of strings; if provided use rendered template strings as keywords
-        description_template: string; optional template string that will be rendered for use as photo description
-        export_db: export database instance compatible with ExportDB_ABC
-        fileutil: file util class compatible with FileUtilABC
-        dry_run: boolean; if True, doesn't actually export or update any files
-        touch_file: boolean; sets file's modification time to match photo date
-        use_photos_export: boolean; if True forces the use of AppleScript to export even if photo not missing
-        convert_to_jpeg: boolean; if True, converts non-jpeg images to jpeg
-        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
-        ignore_date_modified: if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
+        download_missing: attempt download of missing iCloud photos
+        dry_run: bool; if True, doesn't actually export or update any files
+        exiftool_merge_keywords: bool; if True, merged keywords found in file's exif data (requires exiftool)
+        exiftool_merge_persons: bool; if True, merged persons found in file's exif data (requires exiftool)
        exiftool_option: optional list flags (e.g. ["-m", "-F"]) to pass to exiftool
-        exiftool_merge_keywords: boolean; if True, merged keywords found in file's exif data (requires exiftool)
-        exiftool_merge_persons: boolean; if True, merged persons found in file's exif data (requires exiftool)
+        exiftool: bool; use exiftool to write EXIF metadata directly to exported photo
+        export_as_hardlink: bool; hardlink files instead of copying them
+        export_by_date: bool; create export folder in form dest/YYYY/MM/DD
+        export_db: export database instance compatible with ExportDB_ABC
+        export_dir: top-level export directory for {export_dir} template
+        export_edited: bool; if True exports edited version of photo if there is one
+        export_live: bool; also export live video component if photo is a live photo; live video will have same name as photo but with .mov extension
+        export_preview: export the preview image generated by Photos
+        export_raw: bool; if True exports raw image associate with the photo
+        filename_template: template use to determine output file
+        fileutil: file util class compatible with FileUtilABC
+        force_update: bool, only export updated photos but trigger export even if only metadata has changed
+        ignore_date_modified: if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
        jpeg_ext: if not None, specify the extension to use for all JPEG images on export
+        jpeg_quality: float in range 0.0 <= jpeg_quality <= 1.0.  A value of 1.0 specifies use best quality, a value of 0.0 specifies use maximum compression.
+        keyword_template: list of strings; if provided use rendered template strings as keywords
+        num_photos: int, total number of photos that will be exported
+        original_name: bool; use original filename instead of current filename
+        overwrite: bool; overwrite dest file if it already exists
+        person_keyword: bool; if True, exports person names as keywords in metadata
+        photo_num: int, which number photo in total of num_photos is being exported
+        preview_if_missing: bool, export preview if original is missing
+        preview_suffix: str, template to use as suffix for preview images
        replace_keywords: if True, --keyword-template replaces keywords instead of adding keywords
        retry: retry up to retry # of times if there's an error
-        export_dir: top-level export directory for {export_dir} template
-        export_preview: export the preview image generated by Photos
-        preview_suffix: str, template to use as suffix for preview images
-        preview_if_missing: bool, export preview if original is missing
-
+        sidecar_drop_ext: bool; if True, drops photo extension from sidecar name
+        sidecar: list zero, 1 or 2 of ["json","xmp"] of sidecar variety to export
+        skip_original_if_edited: bool; if True does not export original if photo has been edited
+        touch_file: bool; sets file's modification time to match photo date
+        update: bool, only export updated photos
+        use_photos_export: bool; if True forces the use of AppleScript to export even if photo not missing
+        verbose: bool; print verbose output
    Returns:
        list of path(s) of exported photo or None if photo was missing

@@ -2781,7 +2811,7 @@ def export_photo(
            original_filename = str(original_filename)

            verbose_(
-                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename}"
+                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename} ({photo_num}/{num_photos})"
            )

            results += export_photo_to_directory(
@@ -2806,6 +2836,7 @@ def export_photo(
                export_raw=export_raw,
                filename=original_filename,
                fileutil=fileutil,
+                force_update=force_update,
                ignore_date_modified=ignore_date_modified,
                ignore_signature=ignore_signature,
                jpeg_ext=jpeg_ext,
@@ -2918,6 +2949,7 @@ def export_photo(
                    export_raw=not export_original and export_raw,
                    filename=edited_filename,
                    fileutil=fileutil,
+                    force_update=force_update,
                    ignore_date_modified=ignore_date_modified,
                    ignore_signature=ignore_signature,
                    jpeg_ext=jpeg_ext,
@@ -3001,6 +3033,7 @@ def export_photo_to_directory(
    export_raw,
    filename,
    fileutil,
+    force_update,
    ignore_date_modified,
    ignore_signature,
    jpeg_ext,
@@ -3025,53 +3058,24 @@ def export_photo_to_directory(
    """Export photo to directory dest_path"""

    results = ExportResults()
-    # TODO: can be updated to let export do all the missing logic
-    if export_original:
-        if missing and not preview_if_missing:
-            space = " " if not verbose else ""
-            verbose_(
-                f"{space}Skipping missing photo {photo.original_filename} ({photo.uuid})"
-            )
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-        elif (
-            photo.intrash
-            and (not photo.path or (download_missing or use_photos_export))
-            and not preview_if_missing
-        ):
-            # skip deleted files if they're missing or using use_photos_export
-            # as AppleScript/PhotoKit cannot export deleted photos
-            space = " " if not verbose else ""
-            verbose_(
-                f"{space}Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
-            )
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results
-    elif not edited:
-        verbose_(f"Skipping original version of {photo.original_filename}")
+
+    # don't try to export photos in the trash if they're missing
+    photo_path = photo.path if export_original else photo.path_edited
+    if photo.intrash and not photo_path and not preview_if_missing:
+        # skip deleted files if they're missing
+        # as AppleScript/PhotoKit cannot export deleted photos
+        verbose_(
+            f"Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
+        )
+        results.missing.append(str(pathlib.Path(dest_path) / filename))
        return results
-    else:
-        # exporting the edited version
-        if missing and not preview_if_missing:
-            space = " " if not verbose else ""
-            verbose_(f"{space}Skipping missing edited photo for {filename}")
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results
-        elif (
-            photo.intrash
-            and (not photo.path_edited or (download_missing or use_photos_export))
-            and not preview_if_missing
-        ):
-            # skip deleted files if they're missing or using use_photos_export
-            # as AppleScript/PhotoKit cannot export deleted photos
-            space = " " if not verbose else ""
-            verbose_(
-                f"{space}Skipping missing deleted photo {photo.original_filename} ({photo.uuid})"
-            )
-            results.missing.append(str(pathlib.Path(dest_path) / filename))
-            return results

    render_options = RenderOptions(export_dir=export_dir, dest_path=dest_path)

+    if not export_original and not edited:
+        verbose_(f"Skipping original version of {photo.original_filename}")
+        return results
+
    tries = 0
    while tries <= retry:
        tries += 1
@@ -3088,6 +3092,7 @@ def export_photo_to_directory(
                export_as_hardlink=export_as_hardlink,
                export_db=export_db,
                fileutil=fileutil,
+                force_update=force_update,
                ignore_date_modified=ignore_date_modified,
                ignore_signature=ignore_signature,
                jpeg_ext=jpeg_ext,
@@ -3159,7 +3164,7 @@ def export_photo_to_directory(
                )

    if verbose:
-        if update:
+        if update or force_update:
            for new in results.new:
                verbose_(f"Exported new file {new}")
            for updated in results.updated:
@@ -3190,7 +3195,7 @@ def get_filenames_from_template(
    Args:
        photo: a PhotoInfo instance
        filename_template: a PhotoTemplate template string, may be None
-        original_name: boolean; if True, use photo's original filename instead of current filename
+        original_name: bool; if True, use photo's original filename instead of current filename
        dest_path: the path the photo will be exported to
        strip: if True, strips leading/trailing white space from resulting template
        edited: if True, sets {edited_version} field to True, otherwise it gets set to False; set if you want template evaluated for edited version
@@ -3251,9 +3256,9 @@ def get_dirnames_from_template(
    Args:
        photo: a PhotoInstance object
        directory: a PhotoTemplate template string, may be None
-        export_by_date: boolean; if True, creates output directories in form YYYY-MM-DD
+        export_by_date: bool; if True, creates output directories in form YYYY-MM-DD
        dest: top-level destination directory
-        dry_run: boolean; if True, runs in dry-run mode and does not create output directories
+        dry_run: bool; if True, runs in dry-run mode and does not create output directories
        strip: if True, strips leading/trailing white space from resulting template
        edited: if True, sets {edited_version} field to True, otherwise it gets set to False; set if you want template evaluated for edited version

--- a/osxphotos/exiftool.py
+++ b/osxphotos/exiftool.py
@@ -11,6 +11,7 @@ import html
 import json
 import logging
 import os
+import pathlib
 import re
 import shutil
 import subprocess
@@ -19,11 +20,12 @@ from functools import lru_cache  # pylint: disable=syntax-error

 __all__ = [
    "escape_str",
-    "unescape_str",
-    "terminate_exiftool",
-    "get_exiftool_path",
+    "exiftool_can_write",
    "ExifTool",
    "ExifToolCaching",
+    "get_exiftool_path",
+    "terminate_exiftool",
+    "unescape_str",
 ]

 # exiftool -stay_open commands outputs this EOF marker after command is run
@@ -33,6 +35,24 @@ EXIFTOOL_STAYOPEN_EOF_LEN = len(EXIFTOOL_STAYOPEN_EOF)
 # list of exiftool processes to cleanup when exiting or when terminate is called
 EXIFTOOL_PROCESSES = []

+# exiftool supported file types, created by utils/exiftool_supported_types.py
+EXIFTOOL_FILETYPES_JSON = "exiftool_filetypes.json"
+with (pathlib.Path(__file__).parent / EXIFTOOL_FILETYPES_JSON).open("r") as f:
+    EXIFTOOL_SUPPORTED_FILETYPES = json.load(f)
+
+
+def exiftool_can_write(suffix: str) -> bool:
+    """Return True if exiftool supports writing to a file with the given suffix, otherwise False"""
+    if not suffix:
+        return False
+    suffix = suffix.lower()
+    if suffix[0] == ".":
+        suffix = suffix[1:]
+    return (
+        suffix in EXIFTOOL_SUPPORTED_FILETYPES
+        and EXIFTOOL_SUPPORTED_FILETYPES[suffix]["write"]
+    )
+

 def escape_str(s):
    """escape string for use with exiftool -E"""
--- a/osxphotos/exiftool_filetypes.json
+++ b/osxphotos/exiftool_filetypes.json
--- a/osxphotos/export_db.py
+++ b/osxphotos/export_db.py
@@ -10,13 +10,18 @@ import sys
 from abc import ABC, abstractmethod
 from io import StringIO
 from sqlite3 import Error
+from typing import Union

 from ._constants import OSXPHOTOS_EXPORT_DB
 from ._version import __version__
+from .utils import normalize_fs_path

 __all__ = ["ExportDB_ABC", "ExportDBNoOp", "ExportDB", "ExportDBInMemory"]

-OSXPHOTOS_EXPORTDB_VERSION = "4.2"
+OSXPHOTOS_EXPORTDB_VERSION = "5.0"
+OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH = "4.3"
+OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_TABLES = "4.3"
+
 OSXPHOTOS_ABOUT_STRING = f"Created by osxphotos version {__version__} (https://github.com/RhetTbull/osxphotos) on {datetime.datetime.now()}"


@@ -99,6 +104,14 @@ class ExportDB_ABC(ABC):
    def set_detected_text_for_uuid(self, uuid, json_text):
        pass

+    @abstractmethod
+    def set_metadata_for_file(self, filename, metadata):
+        pass
+
+    @abstractmethod
+    def get_metadata_for_file(self, filename):
+        pass
+
    @abstractmethod
    def set_data(
        self,
@@ -110,6 +123,7 @@ class ExportDB_ABC(ABC):
        edited_stat=None,
        info_json=None,
        exif_json=None,
+        metadata=None,
    ):
        pass

@@ -179,6 +193,12 @@ class ExportDBNoOp(ExportDB_ABC):
    def set_detected_text_for_uuid(self, uuid, json_text):
        pass

+    def set_metadata_for_file(self, filename, metadata):
+        pass
+
+    def get_metadata_for_file(self, filename):
+        pass
+
    def set_data(
        self,
        filename,
@@ -189,6 +209,7 @@ class ExportDBNoOp(ExportDB_ABC):
        edited_stat=None,
        info_json=None,
        exif_json=None,
+        metadata=None,
    ):
        pass

@@ -211,12 +232,13 @@ class ExportDB(ExportDB_ABC):
        """query database for filename and return UUID
        returns None if filename not found in database
        """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filepath_normalized = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
            c.execute(
-                "SELECT uuid FROM files WHERE filepath_normalized = ?", (filename,)
+                "SELECT uuid FROM files WHERE filepath_normalized = ?",
+                (filepath_normalized,),
            )
            results = c.fetchone()
            uuid = results[0] if results else None
@@ -228,7 +250,7 @@ class ExportDB(ExportDB_ABC):
    def set_uuid_for_file(self, filename, uuid):
        """set UUID of filename to uuid in the database"""
        filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -245,7 +267,7 @@ class ExportDB(ExportDB_ABC):
        """set stat info for filename
        filename: filename to set the stat info for
        stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        if len(stats) != 3:
            raise ValueError(f"expected 3 elements for stat, got {len(stats)}")

@@ -266,7 +288,7 @@ class ExportDB(ExportDB_ABC):
        """get stat info for filename
        returns: tuple of (mode, size, mtime)
        """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -302,7 +324,7 @@ class ExportDB(ExportDB_ABC):
        """set stat info for filename (after exiftool has updated it)
        filename: filename to set the stat info for
        stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        if len(stats) != 3:
            raise ValueError(f"expected 3 elements for stat, got {len(stats)}")

@@ -323,7 +345,7 @@ class ExportDB(ExportDB_ABC):
        """get stat info for filename (after exiftool has updated it)
        returns: tuple of (mode, size, mtime)
        """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -384,7 +406,7 @@ class ExportDB(ExportDB_ABC):

    def get_exifdata_for_file(self, filename):
        """returns the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -402,7 +424,7 @@ class ExportDB(ExportDB_ABC):

    def set_exifdata_for_file(self, filename, exifdata):
        """sets the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -416,7 +438,7 @@ class ExportDB(ExportDB_ABC):

    def get_sidecar_for_file(self, filename):
        """returns the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -444,7 +466,7 @@ class ExportDB(ExportDB_ABC):

    def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
        """sets the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -502,6 +524,39 @@ class ExportDB(ExportDB_ABC):
        except Error as e:
            logging.warning(e)

+    def set_metadata_for_file(self, filename, metadata):
+        """set metadata of filename in the database"""
+        filename = str(pathlib.Path(filename).relative_to(self._path))
+        filename_normalized = self._normalize_filepath(filename)
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "UPDATE files SET metadata = ? WHERE filepath_normalized = ?;",
+                (metadata, filename_normalized),
+            )
+            conn.commit()
+        except Error as e:
+            logging.warning(e)
+
+    def get_metadata_for_file(self, filename):
+        """get metadata value for file"""
+        filename = self._normalize_filepath_relative(filename)
+        conn = self._conn
+        try:
+            c = conn.cursor()
+            c.execute(
+                "SELECT metadata FROM files WHERE filepath_normalized = ?",
+                (filename,),
+            )
+            results = c.fetchone()
+            metadata = results[0] if results else None
+        except Error as e:
+            logging.warning(e)
+            metadata = None
+
+        return metadata
+
    def set_data(
        self,
        filename,
@@ -512,10 +567,11 @@ class ExportDB(ExportDB_ABC):
        edited_stat=None,
        info_json=None,
        exif_json=None,
+        metadata=None,
    ):
        """sets all the data for file and uuid at once; if any value is None, does not set it"""
        filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
        conn = self._conn
        try:
            c = conn.cursor()
@@ -565,6 +621,15 @@ class ExportDB(ExportDB_ABC):
                    "INSERT OR REPLACE INTO exifdata(filepath_normalized, json_exifdata) VALUES (?, ?);",
                    (filename_normalized, exif_json),
                )
+
+            if metadata is not None:
+                c.execute(
+                    "UPDATE files "
+                    + "SET metadata = ? "
+                    + "WHERE filepath_normalized = ?;",
+                    (metadata, filename_normalized),
+                )
+
            conn.commit()
        except Error as e:
            logging.warning(e)
@@ -577,7 +642,7 @@ class ExportDB(ExportDB_ABC):
            logging.warning(e)

    def _set_stat_for_file(self, table, filename, stats):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        if len(stats) != 3:
            raise ValueError(f"expected 3 elements for stat, got {len(stats)}")

@@ -590,7 +655,7 @@ class ExportDB(ExportDB_ABC):
        conn.commit()

    def _get_stat_for_file(self, table, filename):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
        conn = self._conn
        c = conn.cursor()
        c.execute(
@@ -617,7 +682,7 @@ class ExportDB(ExportDB_ABC):
            conn = self._get_db_connection(dbfile)
            if not conn:
                raise Exception("Error getting connection to database {dbfile}")
-            self._create_db_tables(conn)
+            self._create_or_migrate_db_tables(conn)
            self.was_created = True
            self.was_upgraded = ()
        else:
@@ -625,11 +690,19 @@ class ExportDB(ExportDB_ABC):
            self.was_created = False
            version_info = self._get_database_version(conn)
            if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION:
-                self._create_db_tables(conn)
+                self._create_or_migrate_db_tables(conn)
                self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
            else:
                self.was_upgraded = ()
        self.version = OSXPHOTOS_EXPORTDB_VERSION
+
+        # turn on performance optimizations
+        c = conn.cursor()
+        c.execute("PRAGMA journal_mode=WAL;")
+        c.execute("PRAGMA synchronous=NORMAL;")
+        c.execute("PRAGMA cache_size=-100000;")
+        c.execute("PRAGMA temp_store=MEMORY;")
+
        return conn

    def _get_db_connection(self, dbfile):
@@ -649,104 +722,97 @@ class ExportDB(ExportDB_ABC):
        ).fetchone()
        return (version_info[0], version_info[1])

-    def _create_db_tables(self, conn):
-        """create (if not already created) the necessary db tables for the export database
-        conn: sqlite3 db connection
+    def _create_or_migrate_db_tables(self, conn):
+        """create (if not already created) the necessary db tables for the export database and apply any needed migrations
+
+        Args:
+            conn: sqlite3 db connection
        """
-        sql_commands = {
-            "sql_version_table": """ CREATE TABLE IF NOT EXISTS version (
-                                id INTEGER PRIMARY KEY,
-                                osxphotos TEXT,
-                                exportdb TEXT 
-                                ); """,
-            "sql_about_table": """ CREATE TABLE IF NOT EXISTS about (
-                                id INTEGER PRIMARY KEY,
-                                about TEXT
-                                );""",
-            "sql_files_table": """ CREATE TABLE IF NOT EXISTS files (
-                              id INTEGER PRIMARY KEY,
-                              filepath TEXT NOT NULL,
-                              filepath_normalized TEXT NOT NULL,
-                              uuid TEXT,
-                              orig_mode INTEGER,
-                              orig_size INTEGER,
-                              orig_mtime REAL,
-                              exif_mode INTEGER,
-                              exif_size INTEGER,
-                              exif_mtime REAL
-                              ); """,
-            "sql_files_table_migrate": """ CREATE TABLE IF NOT EXISTS files_migrate (
-                              id INTEGER PRIMARY KEY,
-                              filepath TEXT NOT NULL,
-                              filepath_normalized TEXT NOT NULL,
-                              uuid TEXT,
-                              orig_mode INTEGER,
-                              orig_size INTEGER,
-                              orig_mtime REAL,
-                              exif_mode INTEGER,
-                              exif_size INTEGER,
-                              exif_mtime REAL,
-                              UNIQUE(filepath_normalized)
-                              ); """,
-            "sql_files_migrate": """ INSERT INTO files_migrate SELECT * FROM files;""",
-            "sql_files_drop_tables": """ DROP TABLE files;""",
-            "sql_files_alter": """ ALTER TABLE files_migrate RENAME TO files;""",
-            "sql_runs_table": """ CREATE TABLE IF NOT EXISTS runs (
-                             id INTEGER PRIMARY KEY,
-                             datetime TEXT,
-                             python_path TEXT,
-                             script_name TEXT,
-                             args TEXT,
-                             cwd TEXT 
-                             ); """,
-            "sql_info_table": """ CREATE TABLE IF NOT EXISTS info (
-                             id INTEGER PRIMARY KEY,
-                             uuid text NOT NULL,
-                             json_info JSON 
-                             ); """,
-            "sql_exifdata_table": """ CREATE TABLE IF NOT EXISTS exifdata (
-                             id INTEGER PRIMARY KEY,
-                             filepath_normalized TEXT NOT NULL,
-                             json_exifdata JSON 
-                             ); """,
-            "sql_edited_table": """ CREATE TABLE IF NOT EXISTS edited (
-                              id INTEGER PRIMARY KEY,
-                              filepath_normalized TEXT NOT NULL,
-                              mode INTEGER,
-                              size INTEGER,
-                              mtime REAL
-                              ); """,
-            "sql_converted_table": """ CREATE TABLE IF NOT EXISTS converted (
-                              id INTEGER PRIMARY KEY,
-                              filepath_normalized TEXT NOT NULL,
-                              mode INTEGER,
-                              size INTEGER,
-                              mtime REAL
-                              ); """,
-            "sql_sidecar_table": """ CREATE TABLE IF NOT EXISTS sidecar (
-                              id INTEGER PRIMARY KEY,
-                              filepath_normalized TEXT NOT NULL,
-                              sidecar_data TEXT,
-                              mode INTEGER,
-                              size INTEGER,
-                              mtime REAL
-                              ); """,
-            "sql_detected_text_table": """ CREATE TABLE IF NOT EXISTS detected_text (
-                              id INTEGER PRIMARY KEY,
-                              uuid TEXT NOT NULL,
-                              text_data JSON
-                              ); """,
-            "sql_files_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
-            "sql_info_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
-            "sql_exifdata_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
-            "sql_edited_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
-            "sql_converted_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
-            "sql_sidecar_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
-            "sql_detected_text_idx": """ CREATE UNIQUE INDEX IF NOT EXISTS idx_detected_text on detected_text (uuid);""",
-        }
+        try:
+            version = self._get_database_version(conn)
+        except Exception as e:
+            version = (__version__, OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_TABLES)
+
+        # Current for version 4.3, for anything greater, do a migration after creation
+        sql_commands = [
+            """ CREATE TABLE IF NOT EXISTS version (
+                    id INTEGER PRIMARY KEY,
+                    osxphotos TEXT,
+                    exportdb TEXT 
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS about (
+                    id INTEGER PRIMARY KEY,
+                    about TEXT
+                    );""",
+            """ CREATE TABLE IF NOT EXISTS files (
+                    id INTEGER PRIMARY KEY,
+                    filepath TEXT NOT NULL,
+                    filepath_normalized TEXT NOT NULL,
+                    uuid TEXT,
+                    orig_mode INTEGER,
+                    orig_size INTEGER,
+                    orig_mtime REAL,
+                    exif_mode INTEGER,
+                    exif_size INTEGER,
+                    exif_mtime REAL
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS runs (
+                    id INTEGER PRIMARY KEY,
+                    datetime TEXT,
+                    python_path TEXT,
+                    script_name TEXT,
+                    args TEXT,
+                    cwd TEXT 
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS info (
+                    id INTEGER PRIMARY KEY,
+                    uuid text NOT NULL,
+                    json_info JSON 
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS exifdata (
+                    id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
+                    json_exifdata JSON 
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS edited (
+                    id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
+                    mode INTEGER,
+                    size INTEGER,
+                    mtime REAL
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS converted (
+                    id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
+                    mode INTEGER,
+                    size INTEGER,
+                    mtime REAL
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS sidecar (
+                    id INTEGER PRIMARY KEY,
+                    filepath_normalized TEXT NOT NULL,
+                    sidecar_data TEXT,
+                    mode INTEGER,
+                    size INTEGER,
+                    mtime REAL
+                    ); """,
+            """ CREATE TABLE IF NOT EXISTS detected_text (
+                    id INTEGER PRIMARY KEY,
+                    uuid TEXT NOT NULL,
+                    text_data JSON
+                    ); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_files_filepath_normalized on files (filepath_normalized); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_info_uuid on info (uuid); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_exifdata_filename on exifdata (filepath_normalized); """,
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_edited_filename on edited (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_converted_filename on converted (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_sidecar_filename on sidecar (filepath_normalized);""",
+            """ CREATE UNIQUE INDEX IF NOT EXISTS idx_detected_text on detected_text (uuid);""",
+        ]
+        # create the tables if needed
        try:
            c = conn.cursor()
-            for cmd in sql_commands.values():
+            for cmd in sql_commands:
                c.execute(cmd)
            c.execute(
                "INSERT INTO version(osxphotos, exportdb) VALUES (?, ?);",
@@ -757,6 +823,19 @@ class ExportDB(ExportDB_ABC):
        except Error as e:
            logging.warning(e)

+        # perform needed migrations
+        if version[1] < OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH:
+            self._migrate_normalized_filepath(conn)
+
+        if version[1] < OSXPHOTOS_EXPORTDB_VERSION:
+            try:
+                c = conn.cursor()
+                # add metadata column to files to support --force-update
+                c.execute("ALTER TABLE files ADD COLUMN metadata TEXT;")
+                conn.commit()
+            except Error as e:
+                logging.warning(e)
+
    def __del__(self):
        """ensure the database connection is closed"""
        try:
@@ -782,6 +861,54 @@ class ExportDB(ExportDB_ABC):
        except Error as e:
            logging.warning(e)

+    def _normalize_filepath(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, lower case"""
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _normalize_filepath_relative(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, relative path (to export dir), lower case"""
+        filepath = str(pathlib.Path(filepath).relative_to(self._path))
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _migrate_normalized_filepath(self, conn):
+        """Fix all filepath_normalized columns for unicode normalization"""
+        # Prior to database version 4.3, filepath_normalized was not normalized for unicode
+        c = conn.cursor()
+        migration_sql = [
+            """ CREATE TABLE IF NOT EXISTS files_migrate (
+                    id INTEGER PRIMARY KEY,
+                    filepath TEXT NOT NULL,
+                    filepath_normalized TEXT NOT NULL,
+                    uuid TEXT,
+                    orig_mode INTEGER,
+                    orig_size INTEGER,
+                    orig_mtime REAL,
+                    exif_mode INTEGER,
+                    exif_size INTEGER,
+                    exif_mtime REAL,
+                    UNIQUE(filepath_normalized)
+                    ); """,
+            """ INSERT INTO files_migrate SELECT * FROM files;""",
+            """ DROP TABLE files;""",
+            """ ALTER TABLE files_migrate RENAME TO files;""",
+        ]
+        for sql in migration_sql:
+            c.execute(sql)
+        conn.commit()
+
+        for table in ["converted", "edited", "exifdata", "files", "sidecar"]:
+            old_values = c.execute(
+                f"SELECT filepath_normalized, id FROM {table}"
+            ).fetchall()
+            new_values = [
+                (self._normalize_filepath(filepath_normalized), id_)
+                for filepath_normalized, id_ in old_values
+            ]
+            c.executemany(
+                f"UPDATE {table} SET filepath_normalized=? WHERE id=?", new_values
+            )
+        conn.commit()
+

 class ExportDBInMemory(ExportDB):
    """In memory version of ExportDB
@@ -807,7 +934,7 @@ class ExportDBInMemory(ExportDB):
            conn = self._get_db_connection()
            if not conn:
                raise Exception("Error getting connection to in-memory database")
-            self._create_db_tables(conn)
+            self._create_or_migrate_db_tables(conn)
            self.was_created = True
            self.was_upgraded = ()
        else:
@@ -830,7 +957,7 @@ class ExportDBInMemory(ExportDB):
            self.was_created = False
            _, exportdb_ver = self._get_database_version(conn)
            if exportdb_ver < OSXPHOTOS_EXPORTDB_VERSION:
-                self._create_db_tables(conn)
+                self._create_or_migrate_db_tables(conn)
                self.was_upgraded = (exportdb_ver, OSXPHOTOS_EXPORTDB_VERSION)
            else:
                self.was_upgraded = ()
--- a/osxphotos/photoexporter.py
+++ b/osxphotos/photoexporter.py
@@ -3,7 +3,6 @@


 import dataclasses
-import glob
 import hashlib
 import json
 import logging
@@ -33,7 +32,7 @@ from ._constants import (
 )
 from ._version import __version__
 from .datetime_utils import datetime_tz_to_utc
-from .exiftool import ExifTool
+from .exiftool import ExifTool, exiftool_can_write
 from .export_db import ExportDB_ABC, ExportDBNoOp
 from .fileutil import FileUtil
 from .photokit import (
@@ -45,7 +44,7 @@ from .photokit import (
 )
 from .phototemplate import RenderOptions
 from .uti import get_preferred_uti_extension
-from .utils import increment_filename, increment_filename_with_count, lineno
+from .utils import increment_filename, lineno, list_directory

 __all__ = [
    "ExportError",
@@ -84,6 +83,7 @@ class ExportOptions:
        export_as_hardlink: (bool, default=False): if True, will hardlink files instead of copying them
        export_db: (ExportDB_ABC): instance of a class that conforms to ExportDB_ABC with methods for getting/setting data related to exported files to compare update state
        fileutil: (FileUtilABC): class that conforms to FileUtilABC with various file utilities
+        force_update: (bool, default=False): if True, will export photo if any metadata has changed but export otherwise would not be triggered (e.g. metadata changed but not using exiftool)
        ignore_date_modified (bool): for use with sidecar and exiftool; if True, sets EXIF:ModifyDate to EXIF:DateTimeOriginal even if date_modified is set
        ignore_signature (bool, default=False): ignore file signature when used with update (look only at filename)
        increment (bool, default=True): if True, will increment file name until a non-existant name is found if overwrite=False and increment=False, export will fail if destination file already exists
@@ -129,6 +129,7 @@ class ExportOptions:
    export_as_hardlink: bool = False
    export_db: Optional[ExportDB_ABC] = None
    fileutil: Optional[FileUtil] = None
+    force_update: bool = False
    ignore_date_modified: bool = False
    ignore_signature: bool = False
    increment: bool = True
@@ -450,71 +451,95 @@ class PhotoExporter:
                dest,
                options=options,
            )
+        else:
+            verbose(
+                f"Skipping missing {'edited' if options.edited else 'original'} photo {self.photo.original_filename} ({self.photo.uuid})"
+            )
+            all_results.missing.append(dest)

        # copy live photo associated .mov if requested
-        if (
-            export_original
-            and options.live_photo
-            and self.photo.live_photo
-            and staged_files.original_live
-        ):
+        if export_original and options.live_photo and self.photo.live_photo:
            live_name = dest.parent / f"{dest.stem}.mov"
-            src_live = staged_files.original_live
-            all_results += self._export_photo(
-                src_live,
-                live_name,
-                # don't try to convert the live photo
-                options=dataclasses.replace(options, convert_to_jpeg=False),
-            )
+            if staged_files.original_live:
+                src_live = staged_files.original_live
+                all_results += self._export_photo(
+                    src_live,
+                    live_name,
+                    # don't try to convert the live photo
+                    options=dataclasses.replace(options, convert_to_jpeg=False),
+                )
+            else:
+                verbose(
+                    f"Skipping missing live photo for {self.photo.original_filename} ({self.photo.uuid})"
+                )
+                all_results.missing.append(live_name)

-        if (
-            export_edited
-            and options.live_photo
-            and self.photo.live_photo
-            and staged_files.edited_live
-        ):
+        if export_edited and options.live_photo and self.photo.live_photo:
            live_name = dest.parent / f"{dest.stem}.mov"
-            src_live = staged_files.edited_live
-            all_results += self._export_photo(
-                src_live,
-                live_name,
-                # don't try to convert the live photo
-                options=dataclasses.replace(options, convert_to_jpeg=False),
-            )
+            if staged_files.edited_live:
+                src_live = staged_files.edited_live
+                all_results += self._export_photo(
+                    src_live,
+                    live_name,
+                    # don't try to convert the live photo
+                    options=dataclasses.replace(options, convert_to_jpeg=False),
+                )
+            else:
+                verbose(
+                    f"Skipping missing edited live photo for {self.photo.original_filename} ({self.photo.uuid})"
+                )
+                all_results.missing.append(live_name)

        # copy associated RAW image if requested
-        if options.raw_photo and self.photo.has_raw and staged_files.raw:
-            raw_path = pathlib.Path(staged_files.raw)
-            raw_ext = raw_path.suffix
-            raw_name = dest.parent / f"{dest.stem}{raw_ext}"
-            all_results += self._export_photo(
-                raw_path,
-                raw_name,
-                options=options,
-            )
+        if options.raw_photo and self.photo.has_raw:
+            if staged_files.raw:
+                raw_path = pathlib.Path(staged_files.raw)
+                raw_ext = raw_path.suffix
+                raw_name = dest.parent / f"{dest.stem}{raw_ext}"
+                all_results += self._export_photo(
+                    raw_path,
+                    raw_name,
+                    options=options,
+                )
+            else:
+                # guess at most likely raw name
+                raw_ext = get_preferred_uti_extension(self.photo.uti_raw) or "raw"
+                raw_name = dest.parent / f"{dest.stem}.{raw_ext}"
+                all_results.missing.append(raw_name)
+                verbose(
+                    f"Skipping missing raw photo for {self.photo.original_filename} ({self.photo.uuid})"
+                )

        # copy preview image if requested
-        if options.preview and staged_files.preview:
-            # Photos keeps multiple different derivatives and path_derivatives returns list of them
-            # first derivative is the largest so export that one
-            preview_path = pathlib.Path(staged_files.preview)
-            preview_ext = preview_path.suffix
-            preview_name = (
-                dest.parent / f"{dest.stem}{options.preview_suffix}{preview_ext}"
-            )
-            # if original is missing, the filename won't have been incremented so
-            # need to check here to make sure there aren't duplicate preview files in
-            # the export directory
-            preview_name = (
-                preview_name
-                if options.overwrite or options.update
-                else pathlib.Path(increment_filename(preview_name))
-            )
-            all_results += self._export_photo(
-                preview_path,
-                preview_name,
-                options=options,
-            )
+        if options.preview:
+            if staged_files.preview:
+                # Photos keeps multiple different derivatives and path_derivatives returns list of them
+                # first derivative is the largest so export that one
+                preview_path = pathlib.Path(staged_files.preview)
+                preview_ext = preview_path.suffix
+                preview_name = (
+                    dest.parent / f"{dest.stem}{options.preview_suffix}{preview_ext}"
+                )
+                # if original is missing, the filename won't have been incremented so
+                # need to check here to make sure there aren't duplicate preview files in
+                # the export directory
+                preview_name = (
+                    preview_name
+                    if any([options.overwrite, options.update, options.force_update])
+                    else pathlib.Path(increment_filename(preview_name))
+                )
+                all_results += self._export_photo(
+                    preview_path,
+                    preview_name,
+                    options=options,
+                )
+            else:
+                # don't know what actual preview suffix would be but most likely jpeg
+                preview_name = dest.parent / f"{dest.stem}{options.preview_suffix}.jpeg"
+                all_results.missing.append(preview_name)
+                verbose(
+                    f"Skipping missing preview photo for {self.photo.original_filename} ({self.photo.uuid})"
+                )

        all_results += self._write_sidecar_files(dest=dest, options=options)

@@ -566,7 +591,7 @@ class PhotoExporter:

        # if overwrite==False and #increment==False, export should fail if file exists
        if dest.exists() and not any(
-            [options.increment, options.update, options.overwrite]
+            [options.increment, options.update, options.force_update, options.overwrite]
        ):
            raise FileExistsError(
                f"destination exists ({dest}); overwrite={options.overwrite}, increment={options.increment}"
@@ -578,11 +603,13 @@ class PhotoExporter:
        # e.g. exporting sidecar for file1.png and file1.jpeg
        # if file1.png exists and exporting file1.jpeg,
        # dest will be file1 (1).jpeg even though file1.jpeg doesn't exist to prevent sidecar collision
-        if options.increment and not options.update and not options.overwrite:
+        if options.increment and not any(
+            [options.update, options.force_update, options.overwrite]
+        ):
            return pathlib.Path(increment_filename(dest))

        # if update and file exists, need to check to see if it's the write file by checking export db
-        if options.update and dest.exists() and src:
+        if (options.update or options.force_update) and dest.exists() and src:
            export_db = options.export_db
            fileutil = options.fileutil
            # destination exists, check to see if destination is the right UUID
@@ -598,9 +625,13 @@ class PhotoExporter:
                )
            if dest_uuid != self.photo.uuid:
                # not the right file, find the right one
-                glob_str = str(dest.parent / f"{dest.stem} (*{dest.suffix}")
-                # TODO: use the normalized code in utils
-                dest_files = glob.glob(glob_str)
+                # find files that match "dest_name (*.ext" (e.g. "dest_name (1).jpg", "dest_name (2).jpg)", ...)
+                dest_files = list_directory(
+                    dest.parent,
+                    startswith=f"{dest.stem} (",
+                    endswith=dest.suffix,
+                    include_path=True,
+                )
                for file_ in dest_files:
                    dest_uuid = export_db.get_uuid_for_file(file_)
                    if dest_uuid == self.photo.uuid:
@@ -708,7 +739,7 @@ class PhotoExporter:
        # export live_photo .mov file?
        live_photo = bool(options.live_photo and self.photo.live_photo)

-        overwrite = options.overwrite or options.update
+        overwrite = any([options.overwrite, options.update, options.force_update])

        # figure out which photo version to request
        if options.edited or self.photo.shared:
@@ -816,7 +847,7 @@ class PhotoExporter:

        # export live_photo .mov file?
        live_photo = bool(options.live_photo and self.photo.live_photo)
-        overwrite = options.overwrite or options.update
+        overwrite = any([options.overwrite, options.update, options.force_update])
        edited_version = options.edited or self.photo.shared
        # shared photos (in shared albums) show up as not having adjustments (not edited)
        # but Photos is unable to export the "original" as only a jpeg copy is shared in iCloud
@@ -968,18 +999,16 @@ class PhotoExporter:
        fileutil = options.fileutil
        export_db = options.export_db

-        if options.update:  # updating
+        if options.update or options.force_update:  # updating
            cmp_touch, cmp_orig = False, False
            if dest_exists:
                # update, destination exists, but we might not need to replace it...
-                if options.ignore_signature:
-                    cmp_orig = True
-                    cmp_touch = fileutil.cmp(
-                        src, dest, mtime1=int(self.photo.date.timestamp())
-                    )
-                elif options.exiftool:
+                if options.exiftool:
                    sig_exif = export_db.get_stat_exif_for_file(dest_str)
                    cmp_orig = fileutil.cmp_file_sig(dest_str, sig_exif)
+                    if cmp_orig:
+                        # if signatures match also need to compare exifdata to see if metadata changed
+                        cmp_orig = not self._should_run_exiftool(dest_str, options)
                    sig_exif = (
                        sig_exif[0],
                        sig_exif[1],
@@ -996,10 +1025,17 @@ class PhotoExporter:
                    )
                    cmp_touch = fileutil.cmp_file_sig(dest_str, sig_converted)
                else:
-                    cmp_orig = fileutil.cmp(src, dest)
+                    cmp_orig = options.ignore_signature or fileutil.cmp(src, dest)
                    cmp_touch = fileutil.cmp(
                        src, dest, mtime1=int(self.photo.date.timestamp())
                    )
+                    if options.force_update:
+                        # need to also check the photo's metadata to that in the database
+                        # and if anything changed, we need to update the file
+                        # ony the hex digest of the metadata is stored in the database
+                        photo_digest = hexdigest(self.photo.json())
+                        db_digest = export_db.get_metadata_for_file(dest_str)
+                        cmp_orig = photo_digest == db_digest

                sig_cmp = cmp_touch if options.touch_file else cmp_orig

@@ -1013,7 +1049,7 @@ class PhotoExporter:
                        if sig_edited != (None, None, None)
                        else False
                    )
-                    sig_cmp = sig_cmp and cmp_edited
+                    sig_cmp = sig_cmp and (options.force_update or cmp_edited)

                if (options.export_as_hardlink and dest.samefile(src)) or (
                    not options.export_as_hardlink
@@ -1056,7 +1092,9 @@ class PhotoExporter:
            edited_stat = (
                fileutil.file_sig(src) if options.edited else (None, None, None)
            )
-            if dest_exists and (options.update or options.overwrite):
+            if dest_exists and any(
+                [options.overwrite, options.update, options.force_update]
+            ):
                # need to remove the destination first
                try:
                    fileutil.unlink(dest)
@@ -1099,13 +1137,17 @@ class PhotoExporter:
                        f"Error copying file {src} to {dest_str}: {e} ({lineno(__file__)})"
                    ) from e

+        json_info = self.photo.json()
+        # don't set the metadata digest if not force_update so that future use of force_update catches metadata change
+        metadata_digest = hexdigest(json_info) if options.force_update else None
        export_db.set_data(
            filename=dest_str,
            uuid=self.photo.uuid,
            orig_stat=fileutil.file_sig(dest_str),
            converted_stat=converted_stat,
            edited_stat=edited_stat,
-            info_json=self.photo.json(),
+            info_json=json_info,
+            metadata=metadata_digest,
        )

        return ExportResults(
@@ -1205,10 +1247,13 @@ class PhotoExporter:
                sidecar_filename
            )
            write_sidecar = (
-                not options.update
-                or (options.update and not sidecar_filename.exists())
+                not (options.update or options.force_update)
                or (
-                    options.update
+                    (options.update or options.force_update)
+                    and not sidecar_filename.exists()
+                )
+                or (
+                    (options.update or options.force_update)
                    and (sidecar_digest != old_sidecar_digest)
                    or not fileutil.cmp_file_sig(sidecar_filename, sidecar_sig)
                )
@@ -1256,31 +1301,27 @@ class PhotoExporter:

        exiftool_results = ExportResults()

+        # don't try to write if unsupported file type for exiftool
+        if not exiftool_can_write(os.path.splitext(src)[-1]):
+            exiftool_results.exiftool_warning.append(
+                (
+                    dest,
+                    f"Unsupported file type for exiftool, skipping exiftool for {dest}",
+                )
+            )
+            # set file signature so the file doesn't get re-exported with --update
+            export_db.set_data(
+                dest,
+                uuid=self.photo.uuid,
+                exif_stat=fileutil.file_sig(src),
+                exif_json=self._exiftool_json_sidecar(options=options),
+            )
+            return exiftool_results
+
        # determine if we need to write the exif metadata
        # if we are not updating, we always write
        # else, need to check the database to determine if we need to write
-        run_exiftool = not options.update
-        current_data = "foo"
-        if options.update:
-            files_are_different = False
-            old_data = export_db.get_exifdata_for_file(dest)
-            if old_data is not None:
-                old_data = json.loads(old_data)[0]
-                current_data = json.loads(self._exiftool_json_sidecar(options=options))[
-                    0
-                ]
-                if old_data != current_data:
-                    files_are_different = True
-
-            if old_data is None or files_are_different:
-                # didn't have old data, assume we need to write it
-                # or files were different
-                run_exiftool = True
-            else:
-                verbose(
-                    f"Skipped up to date exiftool metadata for {pathlib.Path(dest).name}"
-                )
-
+        run_exiftool = self._should_run_exiftool(dest, options)
        if run_exiftool:
            verbose(f"Writing metadata with exiftool for {pathlib.Path(dest).name}")
            if not options.dry_run:
@@ -1299,8 +1340,32 @@ class PhotoExporter:
            )
            exiftool_results.exif_updated.append(dest)
            exiftool_results.to_touch.append(dest)
+        else:
+            verbose(
+                f"Skipped up to date exiftool metadata for {pathlib.Path(dest).name}"
+            )
        return exiftool_results

+    def _should_run_exiftool(self, dest, options: ExportOptions) -> bool:
+        """Return True if exiftool should be run to update metadata"""
+        run_exiftool = not (options.update or options.force_update)
+        if options.update or options.force_update:
+            files_are_different = False
+            old_data = options.export_db.get_exifdata_for_file(dest)
+            if old_data is not None:
+                old_data = json.loads(old_data)[0]
+                current_data = json.loads(self._exiftool_json_sidecar(options=options))[
+                    0
+                ]
+                if old_data != current_data:
+                    files_are_different = True
+
+            if old_data is None or files_are_different:
+                # didn't have old data, assume we need to write it
+                # or files were different
+                run_exiftool = True
+        return run_exiftool
+
    def _write_exif_data(self, filepath: str, options: ExportOptions):
        """write exif data to image file at filepath

@@ -1828,7 +1893,7 @@ def _export_photo_uuid_applescript(
        raise ValueError(f"dest {dest} must be a directory")

    if not original ^ edited:
-        raise ValueError(f"edited or original must be True but not both")
+        raise ValueError("edited or original must be True but not both")

    tmpdir = tempfile.TemporaryDirectory(prefix="osxphotos_")

@@ -1851,7 +1916,6 @@ def _export_photo_uuid_applescript(
    if not exported_files or not filename:
        # nothing got exported
        raise ExportError(f"Could not export photo {uuid} ({lineno(__file__)})")
-
    # need to find actual filename as sometimes Photos renames JPG to jpeg on export
    # may be more than one file exported (e.g. if Live Photo, Photos exports both .jpeg and .mov)
    # TemporaryDirectory will cleanup on return
--- a/osxphotos/photoinfo.py
+++ b/osxphotos/photoinfo.py
@@ -54,7 +54,7 @@ from .scoreinfo import ScoreInfo
 from .searchinfo import SearchInfo
 from .text_detection import detect_text
 from .uti import get_preferred_uti_extension, get_uti_for_extension
-from .utils import _debug, _get_resource_loc, findfiles
+from .utils import _debug, _get_resource_loc, list_directory

 __all__ = ["PhotoInfo", "PhotoInfoNone"]

@@ -369,7 +369,7 @@ class PhotoInfo:
        # In Photos 5, raw is in same folder as original but with _4.ext
        # Unless "Copy Items to the Photos Library" is not checked
        # then RAW image is not renamed but has same name is jpeg buth with raw extension
-        # Current implementation uses findfiles to find images with the correct raw UTI extension
+        # Current implementation finds images with the correct raw UTI extension
        # in same folder as the original and with same stem as original in form: original_stem*.raw_ext
        # TODO: I don't like this -- would prefer a more deterministic approach but until I have more
        # data on how Photos stores and retrieves RAW images, this seems to be working
@@ -405,8 +405,7 @@ class PhotoInfo:
            # raw files have same name as original but with _4.raw_ext appended
            # I believe the _4 maps to PHAssetResourceTypeAlternatePhoto = 4
            # see: https://developer.apple.com/documentation/photokit/phassetresourcetype/phassetresourcetypealternatephoto?language=objc
-            glob_str = f"{filestem}_4*"
-            raw_file = findfiles(glob_str, filepath)
+            raw_file = list_directory(filepath, startswith=f"{filestem}_4")
            if not raw_file:
                photopath = None
            else:
@@ -1729,7 +1728,11 @@ class PhotoInfo:
            if isinstance(o, (datetime.date, datetime.datetime)):
                return o.isoformat()

-        return json.dumps(self.asdict(), sort_keys=True, default=default)
+        dict_data = self.asdict()
+        for k, v in dict_data.items():
+            if v and isinstance(v, (list, tuple)) and not isinstance(v[0], dict):
+                dict_data[k] = sorted(v)
+        return json.dumps(dict_data, sort_keys=True, default=default)

    def __eq__(self, other):
        """Compare two PhotoInfo objects for equality"""
--- a/osxphotos/photosdb/photosdb.py
+++ b/osxphotos/photosdb/photosdb.py
@@ -39,6 +39,7 @@ from .._constants import (
    _PHOTOS_5_PROJECT_ALBUM_KIND,
    _PHOTOS_5_ROOT_FOLDER_KIND,
    _PHOTOS_5_SHARED_ALBUM_KIND,
+    _PHOTOS_5_VERSION,
    _TESTED_OS_VERSIONS,
    _UNKNOWN_PERSON,
    BURST_KEY,
@@ -659,14 +660,18 @@ class PhotosDB:

        for person in c:
            pk = person[0]
-            fullname = person[2] if person[2] is not None else _UNKNOWN_PERSON
+            fullname = (
+                normalize_unicode(person[2])
+                if person[2] is not None
+                else _UNKNOWN_PERSON
+            )
            self._dbpersons_pk[pk] = {
                "pk": pk,
                "uuid": person[1],
                "fullname": fullname,
                "facecount": person[3],
                "keyface": person[5],
-                "displayname": person[4],
+                "displayname": normalize_unicode(person[4]),
                "photo_uuid": None,
                "keyface_uuid": None,
            }
@@ -733,13 +738,6 @@ class PhotosDB:
            except KeyError:
                self._dbfaces_pk[pk] = [uuid]

-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
        # Get info on albums
        verbose("Processing albums.")
        c.execute(
@@ -876,14 +874,6 @@ class PhotosDB:
            else:
                self._dbalbum_folders[album] = {}

-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-            logging.debug(pformat(self._dbfolder_details))
-
        # Get info on keywords
        verbose("Processing keywords.")
        c.execute(
@@ -899,13 +889,16 @@ class PhotosDB:
                RKMaster.uuid = RKVersion.masterUuid
            """
        )
-        for keyword in c:
-            if not keyword[1] in self._dbkeywords_uuid:
-                self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword[0] in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword[0]] = []
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword[0]].append(keyword[1])
+        for keyword_title, keyword_uuid, _ in c:
+            keyword_title = normalize_unicode(keyword_title)
+            try:
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
+            except KeyError:
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
+            try:
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
+            except KeyError:
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]

        # Get info on disk volumes
        c.execute("select RKVolume.modelId, RKVolume.name from RKVolume")
@@ -1027,13 +1020,11 @@ class PhotosDB:

        for row in c:
            uuid = row[0]
-            if _debug():
-                logging.debug(f"uuid = '{uuid}, master = '{row[2]}")
            self._dbphotos[uuid] = {}
            self._dbphotos[uuid]["_uuid"] = uuid  # stored here for easier debugging
            self._dbphotos[uuid]["modelID"] = row[1]
            self._dbphotos[uuid]["masterUuid"] = row[2]
-            self._dbphotos[uuid]["filename"] = row[3]
+            self._dbphotos[uuid]["filename"] = normalize_unicode(row[3])

            # There are sometimes negative values for lastmodifieddate in the database
            # I don't know what these mean but they will raise exception in datetime if
@@ -1272,13 +1263,13 @@ class PhotosDB:
            info["volumeId"] = row[1]
            info["imagePath"] = row[2]
            info["isMissing"] = row[3]
-            info["originalFilename"] = row[4]
+            info["originalFilename"] = normalize_unicode(row[4])
            info["UTI"] = row[5]
            info["modelID"] = row[6]
            info["fileSize"] = row[7]
            info["isTrulyRAW"] = row[8]
            info["alternateMasterUuid"] = row[9]
-            info["filename"] = row[10]
+            info["filename"] = normalize_unicode(row[10])
            self._dbphotos_master[uuid] = info

        # get details needed to find path of the edited photos
@@ -1550,39 +1541,6 @@ class PhotosDB:

        # done processing, dump debug data if requested
        verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))

    def _build_album_folder_hierarchy_4(self, uuid, folders=None):
        """recursively build folder/album hierarchy
@@ -1673,7 +1631,7 @@ class PhotosDB:
        for person in c:
            pk = person[0]
            fullname = (
-                person[2]
+                normalize_unicode(person[2])
                if (person[2] != "" and person[2] is not None)
                else _UNKNOWN_PERSON
            )
@@ -1683,7 +1641,7 @@ class PhotosDB:
                "fullname": fullname,
                "facecount": person[3],
                "keyface": person[4],
-                "displayname": person[5],
+                "displayname": normalize_unicode(person[5]),
                "photo_uuid": None,
                "keyface_uuid": None,
            }
@@ -1747,13 +1705,6 @@ class PhotosDB:
            except KeyError:
                self._dbfaces_pk[pk] = [uuid]

-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
        # get details about albums
        verbose("Processing albums.")
        c.execute(
@@ -1870,13 +1821,6 @@ class PhotosDB:
                # shared albums can't be in folders
                self._dbalbum_folders[album] = []

-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-
        # get details on keywords
        verbose("Processing keywords.")
        c.execute(
@@ -1886,29 +1830,22 @@ class PhotosDB:
                JOIN Z_1KEYWORDS ON Z_1KEYWORDS.Z_1ASSETATTRIBUTES = ZADDITIONALASSETATTRIBUTES.Z_PK 
                JOIN ZKEYWORD ON ZKEYWORD.Z_PK = {keyword_join} """
        )
-        for keyword in c:
-            keyword_title = normalize_unicode(keyword[0])
-            if not keyword[1] in self._dbkeywords_uuid:
-                self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword_title in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword_title] = []
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword_title].append(keyword[1])
-
-        if _debug():
-            logging.debug(f"Finished walking through keywords")
-            logging.debug(pformat(self._dbkeywords_keyword))
-            logging.debug(pformat(self._dbkeywords_uuid))
+        for keyword_title, keyword_uuid in c:
+            keyword_title = normalize_unicode(keyword_title)
+            try:
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
+            except KeyError:
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
+            try:
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
+            except KeyError:
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]

        # get details on disk volumes
        c.execute("SELECT ZUUID, ZNAME from ZFILESYSTEMVOLUME")
        for vol in c:
            self._dbvolumes[vol[0]] = vol[1]

-        if _debug():
-            logging.debug(f"Finished walking through volumes")
-            logging.debug(self._dbvolumes)
-
        # get details about photos
        verbose("Processing photo details.")
        c.execute(
@@ -2042,8 +1979,8 @@ class PhotosDB:

            info["hidden"] = row[9]
            info["favorite"] = row[10]
-            info["originalFilename"] = row[3]
-            info["filename"] = row[12]
+            info["originalFilename"] = normalize_unicode(row[3])
+            info["filename"] = normalize_unicode(row[12])
            info["directory"] = row[11]

            # set latitude and longitude
@@ -2521,48 +2458,6 @@ class PhotosDB:

        # done processing, dump debug data if requested
        verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Album folders (_dbalbum_folders):")
-            logging.debug(pformat(self._dbalbum_folders))
-
-            logging.debug("Album parent folders (_dbalbum_parent_folders):")
-            logging.debug(pformat(self._dbalbum_parent_folders))
-
-            logging.debug("Albums pk (_dbalbums_pk):")
-            logging.debug(pformat(self._dbalbums_pk))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))

    def _process_moments(self):
        """Process data from ZMOMENT table"""
@@ -2623,8 +2518,8 @@ class PhotosDB:
            moment_info["modificationDate"] = row[6]
            moment_info["representativeDate"] = row[7]
            moment_info["startDate"] = row[8]
-            moment_info["subtitle"] = row[9]
-            moment_info["title"] = row[10]
+            moment_info["subtitle"] = normalize_unicode(row[9])
+            moment_info["title"] = normalize_unicode(row[10])
            moment_info["uuid"] = row[11]

            # if both lat/lon == -180, then it means location undefined
@@ -3027,6 +2922,7 @@ class PhotosDB:
            if keywords:
                keyword_set = set()
                for keyword in keywords:
+                    keyword = normalize_unicode(keyword)
                    if keyword in self._dbkeywords_keyword:
                        keyword_set.update(self._dbkeywords_keyword[keyword])
                photos_sets.append(keyword_set)
@@ -3034,6 +2930,7 @@ class PhotosDB:
            if persons:
                person_set = set()
                for person in persons:
+                    person = normalize_unicode(person)
                    if person in self._dbpersons_fullname:
                        for pk in self._dbpersons_fullname[person]:
                            try:
@@ -3076,8 +2973,6 @@ class PhotosDB:
                ):
                    info = PhotoInfo(db=self, uuid=p, info=self._dbphotos[p])
                    photoinfo.append(info)
-        if _debug:
-            logging.debug(f"photoinfo: {pformat(photoinfo)}")

        return photoinfo

@@ -3414,23 +3309,35 @@ class PhotosDB:
                # case-insensitive
                for n in name:
                    n = n.lower()
-                    photo_list.extend(
-                        [
-                            p
-                            for p in photos
-                            if n in p.filename.lower()
-                            or n in p.original_filename.lower()
-                        ]
-                    )
+                    if self._db_version >= _PHOTOS_5_VERSION:
+                        # search only original_filename (#594)
+                        photo_list.extend(
+                            [p for p in photos if n in p.original_filename.lower()]
+                        )
+                    else:
+                        photo_list.extend(
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename.lower()
+                                or n in p.original_filename.lower()
+                            ]
+                        )
            else:
                for n in name:
-                    photo_list.extend(
-                        [
-                            p
-                            for p in photos
-                            if n in p.filename or n in p.original_filename
-                        ]
-                    )
+                    if self._db_version >= _PHOTOS_5_VERSION:
+                        # search only original_filename (#594)
+                        photo_list.extend(
+                            [p for p in photos if n in p.original_filename]
+                        )
+                    else:
+                        photo_list.extend(
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename or n in p.original_filename
+                            ]
+                        )
            photos = photo_list

        if options.min_size:
--- a/osxphotos/searchinfo.py
+++ b/osxphotos/searchinfo.py
@@ -211,10 +211,12 @@ class SearchInfo:
        """return list of text for a specified category ID"""
        if self._db_searchinfo:
            content = "normalized_string" if self._normalized else "content_string"
-            return [
-                rec[content]
-                for rec in self._db_searchinfo
-                if rec["category"] == category
-            ]
+            return sorted(
+                [
+                    rec[content]
+                    for rec in self._db_searchinfo
+                    if rec["category"] == category
+                ]
+            )
        else:
            return []
--- a/osxphotos/utils.py
+++ b/osxphotos/utils.py
@@ -1,5 +1,6 @@
 """ Utility functions used in osxphotos """

+import datetime
 import fnmatch
 import glob
 import importlib
@@ -16,7 +17,7 @@ import sys
 import unicodedata
 import urllib.parse
 from plistlib import load as plistload
-from typing import Callable, List, Union
+from typing import Callable, List, Union, Optional

 import CoreFoundation
 import objc
@@ -27,7 +28,6 @@ from ._constants import UNICODE_FORMAT
 __all__ = [
    "dd_to_dms_str",
    "expand_and_validate_filepath",
-    "findfiles",
    "get_last_library_path",
    "get_system_library_path",
    "increment_filename_with_count",
@@ -265,7 +265,9 @@ def list_photo_libraries():
    # On older MacOS versions, mdfind appears to ignore some libraries
    # glob to find libraries in ~/Pictures then mdfind to find all the others
    # TODO: make this more robust
-    lib_list = glob.glob(f"{pathlib.Path.home()}/Pictures/*.photoslibrary")
+    lib_list = list_directory(
+        f"{pathlib.Path.home()}/Pictures/", glob="*.photoslibrary"
+    )

    # On older OS, may not get all libraries so make sure we get the last one
    last_lib = get_last_library_path()
@@ -289,27 +291,90 @@ def normalize_fs_path(path: str) -> str:
    return unicodedata.normalize("NFD", path)


-def findfiles(pattern, path):
-    """Returns list of filenames from path matched by pattern
-    shell pattern. Matching is case-insensitive.
-    If 'path_' is invalid/doesn't exist, returns []."""
-    if not os.path.isdir(path):
+# def findfiles(pattern, path):
+#     """Returns list of filenames from path matched by pattern
+#     shell pattern. Matching is case-insensitive.
+#     If 'path_' is invalid/doesn't exist, returns []."""
+#     if not os.path.isdir(path):
+#         return []
+
+#     # paths need to be normalized for unicode as filesystem returns unicode in NFD form
+#     pattern = normalize_fs_path(pattern)
+#     rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
+#     files = os.listdir(path)
+#     return [name for name in files if rule.match(name)]
+
+
+def list_directory(
+    directory: Union[str, pathlib.Path],
+    startswith: Optional[str] = None,
+    endswith: Optional[str] = None,
+    contains: Optional[str] = None,
+    glob: Optional[str] = None,
+    include_path: bool = False,
+    case_sensitive: bool = False,
+) -> List[Union[str, pathlib.Path]]:
+    """List directory contents and return list of files or directories matching search criteria.
+    Accounts for case-insensitive filesystems, unicode filenames. directory can be a str or a pathlib.Path object.
+
+    Args:
+        directory: directory to search
+        startswith: string to match at start of filename
+        endswith: string to match at end of filename
+        contains: string to match anywhere in filename
+        glob: shell-style glob pattern to match filename
+        include_path: if True, return full path to file
+        case_sensitive: if True, match case-sensitively
+
+    Returns: List of files or directories matching search criteria as either str or pathlib.Path objects depending on the input type;
+    returns empty list if directory is invalid or doesn't exist.
+
+    """
+    is_pathlib = isinstance(directory, pathlib.Path)
+    if is_pathlib:
+        directory = str(directory)
+
+    if not os.path.isdir(directory):
        return []

-    # paths need to be normalized for unicode as filesystem returns unicode in NFD form
-    pattern = normalize_fs_path(pattern)
-    rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
-    files = os.listdir(path)
-    return [name for name in files if rule.match(name)]
+    startswith = normalize_fs_path(startswith) if startswith else None
+    endswith = normalize_fs_path(endswith) if endswith else None
+    contains = normalize_fs_path(contains) if contains else None
+    glob = normalize_fs_path(glob) if glob else None

+    files = [normalize_fs_path(f) for f in os.listdir(directory)]
+    if not case_sensitive:
+        files_normalized = {f.lower(): f for f in files}
+        files = [f.lower() for f in files]
+        startswith = startswith.lower() if startswith else None
+        endswith = endswith.lower() if endswith else None
+        contains = contains.lower() if contains else None
+        glob = glob.lower() if glob else None
+    else:
+        files_normalized = {f: f for f in files}

-def list_directory_startswith(directory_path: str, startswith: str) -> List[str]:
-    """List directory contents and return list of files starting with startswith; returns [] if directory doesn't exist"""
-    if not os.path.isdir(directory_path):
-        return []
-    startswith = normalize_fs_path(startswith)
-    files = [normalize_fs_path(f) for f in os.listdir(directory_path)]
-    return [f for f in files if f.startswith(startswith)]
+    if startswith:
+        files = [f for f in files if f.startswith(startswith)]
+    if endswith:
+        endswith = normalize_fs_path(endswith)
+        files = [f for f in files if f.endswith(endswith)]
+    if contains:
+        contains = normalize_fs_path(contains)
+        files = [f for f in files if contains in f]
+    if glob:
+        glob = normalize_fs_path(glob)
+        flags = re.IGNORECASE if not case_sensitive else 0
+        rule = re.compile(fnmatch.translate(glob), flags)
+        files = [f for f in files if rule.match(f)]
+
+    files = [files_normalized[f] for f in files]
+
+    if include_path:
+        files = [os.path.join(directory, f) for f in files]
+    if is_pathlib:
+        files = [pathlib.Path(f) for f in files]
+
+    return files


 def _open_sql_file(dbname):
@@ -380,8 +445,8 @@ def increment_filename_with_count(
    Note: This obviously is subject to race condition so using with caution.
    """
    dest = filepath if isinstance(filepath, pathlib.Path) else pathlib.Path(filepath)
-    dest_files = list_directory_startswith(str(dest.parent), dest.stem)
-    dest_files = [pathlib.Path(f).stem.lower() for f in dest_files]
+    dest_files = list_directory(dest.parent, startswith=dest.stem)
+    dest_files = [f.stem.lower() for f in dest_files]
    dest_new = f"{dest.stem} ({count})" if count else dest.stem
    dest_new = normalize_fs_path(dest_new)

@@ -447,3 +512,9 @@ def load_function(pyfile: str, function_name: str) -> Callable:
        sys.path = syspath

    return func
+
+
+def format_sec_to_hhmmss(sec: float) -> str:
+    """Format seconds to hh:mm:ss"""
+    delta = datetime.timedelta(seconds=sec)
+    return str(delta).split(".")[0]
--- a/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite
+++ b/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite
--- a/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock
+++ b/tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock
@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>585B80BF-8D1F-55EF-A9E8-6CF4E5523959</string>
 	<key>pid</key>
-	<integer>1961</integer>
+	<integer>14817</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>
--- a/tests/Test-10.15.7.photoslibrary/database/search/psi.sqlite
+++ b/tests/Test-10.15.7.photoslibrary/database/search/psi.sqlite
--- a/tests/Test-10.15.7.photoslibrary/database/search/synonymsProcess.plist
+++ b/tests/Test-10.15.7.photoslibrary/database/search/synonymsProcess.plist
--- a/tests/Test-10.15.7.photoslibrary/database/search/zeroKeywords.data
+++ b/tests/Test-10.15.7.photoslibrary/database/search/zeroKeywords.data
--- a/tests/Test-10.15.7.photoslibrary/originals/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/originals/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A.jpeg
--- a/tests/Test-10.15.7.photoslibrary/originals/5/54E76FCB-D353-4557-9997-0A457BCB4D48.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/originals/5/54E76FCB-D353-4557-9997-0A457BCB4D48.jpeg
--- a/tests/Test-10.15.7.photoslibrary/originals/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/originals/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091.jpeg
--- a/tests/Test-10.15.7.photoslibrary/originals/F/F207D5DE-EFAD-4217-8424-0764AAC971D0.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/originals/F/F207D5DE-EFAD-4217-8424-0764AAC971D0.jpeg
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.AOI.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.AOI.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.AOI.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.AOI.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.Nature.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.Nature.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.Nature.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.Nature.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.POI.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.POI.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.POI.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.POI.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSBusinessCategoryCache.ROI.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSLocationCache.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSLocationCache.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSLocationCache.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSLocationCache.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSPublicEventCache.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSPublicEventCache.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSPublicEventCache.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/CLSPublicEventCache.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGCurationCache.sqlite.sqlite-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGCurationCache.sqlite.sqlite-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGCurationCache.sqlite.sqlite-wal
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGCurationCache.sqlite.sqlite-wal
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGSearchComputationCache.plist
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PGSearchComputationCache.plist
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist
@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2021-09-14T04:40:41Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
 	<date>2021-07-20T05:48:08Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
 	<date>2021-07-20T05:47:59Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2021-09-14T04:40:43Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 </dict>
 </plist>
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotosGraph/photosgraph.kgdb
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotosGraph/photosgraph.kgdb
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotosGraph/photosgraph.kgdb-shm
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotosGraph/photosgraph.kgdb-shm
--- a/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/changetoken.plist
+++ b/tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/changetoken.plist
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A_1_105_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A_1_105_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/5/54E76FCB-D353-4557-9997-0A457BCB4D48_1_105_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/5/54E76FCB-D353-4557-9997-0A457BCB4D48_1_105_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091_1_105_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091_1_105_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/F/F207D5DE-EFAD-4217-8424-0764AAC971D0_1_105_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/F/F207D5DE-EFAD-4217-8424-0764AAC971D0_1_105_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A_4_5005_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/2/2DFD33F1-A5D8-486F-A3A9-98C07995535A_4_5005_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/5/54E76FCB-D353-4557-9997-0A457BCB4D48_4_5005_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/5/54E76FCB-D353-4557-9997-0A457BCB4D48_4_5005_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091_4_5005_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/7/7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091_4_5005_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/F/F207D5DE-EFAD-4217-8424-0764AAC971D0_4_5005_c.jpeg
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/masters/F/F207D5DE-EFAD-4217-8424-0764AAC971D0_4_5005_c.jpeg
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/3305.ithmb
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/3305.ithmb
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/4031.ithmb
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/4031.ithmb
--- a/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/4132.ithmb
+++ b/tests/Test-10.15.7.photoslibrary/resources/derivatives/thumbs/4132.ithmb
--- a/tests/Test-10.15.7.photoslibrary/resources/journals/Album-change.plj
+++ b/tests/Test-10.15.7.photoslibrary/resources/journals/Album-change.plj
--- a/tests/Test-10.15.7.photoslibrary/resources/journals/Asset-change.plj
+++ b/tests/Test-10.15.7.photoslibrary/resources/journals/Asset-change.plj
--- a/tests/Test-10.15.7.photoslibrary/resources/journals/Folder-change.plj
+++ b/tests/Test-10.15.7.photoslibrary/resources/journals/Folder-change.plj
--- a/tests/Test-10.15.7.photoslibrary/resources/journals/HistoryToken.plist
+++ b/tests/Test-10.15.7.photoslibrary/resources/journals/HistoryToken.plist
--- a/tests/Test-10.15.7.photoslibrary/resources/journals/ImportSession-change.plj
+++ b/tests/Test-10.15.7.photoslibrary/resources/journals/ImportSession-change.plj
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -40,7 +40,7 @@ else:

@pytest.fixture(autouse=True)
 def reset_singletons():
-    """ Need to clean up any ExifTool singletons between tests """
+    """Need to clean up any ExifTool singletons between tests"""
    _ExifToolProc.instance = None


@@ -73,7 +73,7 @@ def pytest_collection_modifyitems(config, items):


 def copy_photos_library(photos_library=TEST_LIBRARY, delay=0):
-    """ copy the test library and open Photos, returns path to copied library """
+    """copy the test library and open Photos, returns path to copied library"""
    script = AppleScript(
        """
        tell application "Photos"
@@ -118,3 +118,9 @@ def copy_photos_library(photos_library=TEST_LIBRARY, delay=0):
@pytest.fixture
 def addalbum_library():
    copy_photos_library(delay=10)
+
+
+def copy_photos_library_to_path(photos_library_path: str, dest_path: str) -> str:
+    """Copy a photos library to a folder"""
+    ditto(photos_library_path, dest_path)
+    return dest_path
--- a/tests/search_info_test_data_10_15_7.json
+++ b/tests/search_info_test_data_10_15_7.json
--- a/tests/test_albums_folders_catalina_10_15_7.py
+++ b/tests/test_albums_folders_catalina_10_15_7.py
@@ -21,6 +21,7 @@ FOLDER_ALBUM_DICT = {
 ALBUM_NAMES = [
    "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
    "2019-10/11 Paris Clermont",
+    "Água",
    "AlbumInFolder",
    "EmptyAlbum",
    "I have a deleted twin",
@@ -38,6 +39,7 @@ ALBUM_NAMES = [
 ALBUM_PARENT_DICT = {
    "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": None,
    "2019-10/11 Paris Clermont": None,
+    "Água": None,
    "AlbumInFolder": "SubFolder2",
    "EmptyAlbum": None,
    "I have a deleted twin": None,
@@ -54,6 +56,7 @@ ALBUM_PARENT_DICT = {
 ALBUM_FOLDER_NAMES_DICT = {
    "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": [],
    "2019-10/11 Paris Clermont": [],
+    "Água": [],
    "AlbumInFolder": ["Folder1", "SubFolder2"],
    "EmptyAlbum": [],
    "I have a deleted twin": [],
@@ -70,6 +73,7 @@ ALBUM_FOLDER_NAMES_DICT = {
 ALBUM_LEN_DICT = {
    "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": 1,
    "2019-10/11 Paris Clermont": 1,
+    "Água": 3,
    "AlbumInFolder": 2,
    "EmptyAlbum": 0,
    "I have a deleted twin": 1,
@@ -103,6 +107,11 @@ ALBUM_PHOTO_UUID_DICT = {
        "4D521201-92AC-43E5-8F7C-59BC41C37A96",
        "8E1D7BC9-9321-44F9-8CFB-4083F6B9232A",
    ],
+    "Água": [
+        "7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091",
+        "2DFD33F1-A5D8-486F-A3A9-98C07995535A",
+        "54E76FCB-D353-4557-9997-0A457BCB4D48",
+    ],
 }

 UUID_DICT = {
--- a/tests/test_catalina_10_15_7.py
+++ b/tests/test_catalina_10_15_7.py
@@ -24,10 +24,10 @@ PHOTOS_DB = "tests/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_DB_PATH = "/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_LIBRARY_PATH = "/Test-10.15.7.photoslibrary"

-PHOTOS_DB_LEN = 25
-PHOTOS_NOT_IN_TRASH_LEN = 23
+PHOTOS_DB_LEN = 29
+PHOTOS_NOT_IN_TRASH_LEN = 27
 PHOTOS_IN_TRASH_LEN = 2
-PHOTOS_DB_IMPORT_SESSIONS = 17
+PHOTOS_DB_IMPORT_SESSIONS = 21

 KEYWORDS = [
    "Kids",
@@ -72,6 +72,7 @@ ALBUMS = [
    "Sorted Oldest First",
    "Sorted Title",
    "Test Album",  # there are 2 albums named "Test Album" for testing duplicate album names
+    "Água",
 ]
 KEYWORDS_DICT = {
    "Drink": 2,
@@ -115,6 +116,7 @@ ALBUM_DICT = {
    "Sorted Oldest First": 3,
    "Sorted Title": 3,
    "Test Album": 2,
+    "Água": 3,
 }  # Note: there are 2 albums named "Test Album" for testing duplicate album names

 UUID_DICT = {
@@ -1091,7 +1093,7 @@ def test_from_to_date(photosdb):
    time.tzset()

    photos = photosdb.photos(from_date=datetime.datetime(2018, 10, 28))
-    assert len(photos) == 16
+    assert len(photos) == 20

    photos = photosdb.photos(to_date=datetime.datetime(2018, 10, 28))
    assert len(photos) == 7
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
--- a/tests/test_export_db.py
+++ b/tests/test_export_db.py
@@ -7,6 +7,7 @@ import pytest
 EXIF_DATA = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "EXIF:ImageDescription": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "XMP:Title": "Elder Park", "EXIF:GPSLatitude": "34 deg 55' 8.01\" S", "EXIF:GPSLongitude": "138 deg 35' 48.70\" E", "Composite:GPSPosition": "34 deg 55' 8.01\" S, 138 deg 35' 48.70\" E", "EXIF:GPSLatitudeRef": "South", "EXIF:GPSLongitudeRef": "East", "EXIF:DateTimeOriginal": "2017:06:20 17:18:56", "EXIF:OffsetTimeOriginal": "+09:30", "EXIF:ModifyDate": "2020:05:18 14:42:04"}]"""
 INFO_DATA = """{"uuid": "3DD2C897-F19E-4CA6-8C22-B027D5A71907", "filename": "3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "original_filename": "IMG_4547.jpg", "date": "2017-06-20T17:18:56.518000+09:30", "description": "\u2068Elder Park\u2069, \u2068Adelaide\u2069, \u2068Australia\u2069", "title": "Elder Park", "keywords": [], "labels": ["Statue", "Art"], "albums": ["AlbumInFolder"], "folders": {"AlbumInFolder": ["Folder1", "SubFolder2"]}, "persons": [], "path": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/originals/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg", "ismissing": false, "hasadjustments": true, "external_edit": false, "favorite": false, "hidden": false, "latitude": -34.91889167000001, "longitude": 138.59686167, "path_edited": "/Users/rhet/Pictures/Test-10.15.4.photoslibrary/resources/renders/3/3DD2C897-F19E-4CA6-8C22-B027D5A71907_1_201_a.jpeg", "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": false, "incloud": null, "date_modified": "2020-05-18T14:42:04.608664+09:30", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Elder Park, Adelaide, South Australia, Australia, River Torrens", "names": {"field0": [], "country": ["Australia"], "state_province": ["South Australia"], "sub_administrative_area": ["Adelaide"], "city": ["Adelaide", "Adelaide"], "field5": [], "additional_city_info": ["Adelaide CBD", "Tarndanya"], "ocean": [], "area_of_interest": ["Elder Park", ""], "inland_water": ["River Torrens", "River Torrens"], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": ["River Torrens", "River Torrens"]}, "country_code": "AU", "ishome": false, "address_str": "River Torrens, Adelaide SA, Australia", "address": {"street": null, "sub_locality": "Tarndanya", "city": "Adelaide", "sub_administrative_area": "Adelaide", "state_province": "SA", "postal_code": null, "country": "Australia", "iso_country_code": "AU"}}, "exif": {"flash_fired": false, "iso": 320, "metering_mode": 3, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.058823529411764705, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
 SIDECAR_DATA = """FOO_BAR"""
+METADATA_DATA = "FIZZ"

 EXIF_DATA2 = """[{"_CreatedBy": "osxphotos, https://github.com/RhetTbull/osxphotos", "XMP:Title": "St. James's Park", "XMP:TagsList": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "IPTC:Keywords": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "XMP:Subject": ["London 2018", "St. James's Park", "England", "United Kingdom", "UK", "London"], "EXIF:GPSLatitude": "51 deg 30' 12.86\" N", "EXIF:GPSLongitude": "0 deg 7' 54.50\" W", "Composite:GPSPosition": "51 deg 30' 12.86\" N, 0 deg 7' 54.50\" W", "EXIF:GPSLatitudeRef": "North", "EXIF:GPSLongitudeRef": "West", "EXIF:DateTimeOriginal": "2018:10:13 09:18:12", "EXIF:OffsetTimeOriginal": "-04:00", "EXIF:ModifyDate": "2019:12:08 14:06:44"}]"""
 INFO_DATA2 = """{"uuid": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529", "filename": "F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "original_filename": "IMG_8440.JPG", "date": "2019-06-11T11:42:06.711805-07:00", "description": null, "title": null, "keywords": [], "labels": ["Sky", "Cloudy", "Fence", "Land", "Outdoor", "Park", "Amusement Park", "Roller Coaster"], "albums": [], "folders": {}, "persons": [], "path": "/Volumes/MacBook Catalina - Data/Users/rhet/Pictures/Photos Library.photoslibrary/originals/F/F2BB3F98-90F0-4E4C-A09B-25C6822A4529.jpeg", "ismissing": false, "hasadjustments": false, "external_edit": false, "favorite": false, "hidden": false, "latitude": 33.81558666666667, "longitude": -117.99298, "path_edited": null, "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": true, "incloud": true, "date_modified": "2019-10-14T00:51:47.141950-07:00", "portrait": false, "screenshot": false, "slow_mo": false, "time_lapse": false, "hdr": false, "selfie": false, "panorama": false, "has_raw": false, "uti_raw": null, "path_raw": null, "place": {"name": "Adventure City, Stanton, California, United States", "names": {"field0": [], "country": ["United States"], "state_province": ["California"], "sub_administrative_area": ["Orange"], "city": ["Stanton", "Anaheim", "Anaheim"], "field5": [], "additional_city_info": ["West Anaheim"], "ocean": [], "area_of_interest": ["Adventure City", "Adventure City"], "inland_water": [], "field10": [], "region": [], "sub_throughfare": [], "field13": [], "postal_code": [], "field15": [], "field16": [], "street_address": [], "body_of_water": []}, "country_code": "US", "ishome": false, "address_str": "Adventure City, 1240 S Beach Blvd, Anaheim, CA  92804, United States", "address": {"street": "1240 S Beach Blvd", "sub_locality": "West Anaheim", "city": "Stanton", "sub_administrative_area": "Orange", "state_province": "CA", "postal_code": "92804", "country": "United States", "iso_country_code": "US"}}, "exif": {"flash_fired": false, "iso": 25, "metering_mode": 5, "sample_rate": null, "track_format": null, "white_balance": 0, "aperture": 2.2, "bit_rate": null, "duration": null, "exposure_bias": 0.0, "focal_length": 4.15, "fps": null, "latitude": null, "longitude": null, "shutter_speed": 0.0004940711462450593, "camera_make": "Apple", "camera_model": "iPhone 6s", "codec": null, "lens_model": "iPhone 6s back camera 4.15mm f/2.2"}}"""
@@ -64,6 +65,7 @@ def test_export_db():
        (10, 11, 12),
        INFO_DATA,
        EXIF_DATA,
+        METADATA_DATA,
    )
    assert db.get_uuid_for_file(filepath2) == "BAR-FOO"
    assert db.get_info_for_uuid("BAR-FOO") == INFO_DATA
@@ -73,6 +75,7 @@ def test_export_db():
    assert db.get_stat_converted_for_file(filepath2) == (7, 8, 9)
    assert db.get_stat_edited_for_file(filepath2) == (10, 11, 12)
    assert sorted(db.get_previous_uuids()) == (["BAR-FOO", "FOO-BAR"])
+    assert db.get_metadata_for_file(filepath2) == METADATA_DATA

    # test set_data value=None doesn't overwrite existing data
    db.set_data(
@@ -84,6 +87,7 @@ def test_export_db():
        None,
        None,
        None,
+        None,
    )
    assert db.get_uuid_for_file(filepath2) == "BAR-FOO"
    assert db.get_info_for_uuid("BAR-FOO") == INFO_DATA
@@ -93,6 +97,7 @@ def test_export_db():
    assert db.get_stat_converted_for_file(filepath2) == (7, 8, 9)
    assert db.get_stat_edited_for_file(filepath2) == (10, 11, 12)
    assert sorted(db.get_previous_uuids()) == (["BAR-FOO", "FOO-BAR"])
+    assert db.get_metadata_for_file(filepath2) == METADATA_DATA

    # close and re-open
    db.close()
@@ -107,6 +112,8 @@ def test_export_db():
    assert db.get_stat_edited_for_file(filepath2) == (10, 11, 12)
    assert sorted(db.get_previous_uuids()) == (["BAR-FOO", "FOO-BAR"])
    assert json.loads(db.get_detected_text_for_uuid("FOO-BAR")) == [["foo", 0.5]]
+    assert db.get_metadata_for_file(filepath2) == METADATA_DATA
+

    # update data
    db.set_uuid_for_file(filepath, "FUBAR")
@@ -148,9 +155,10 @@ def test_export_db_no_op():
    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
    assert db.get_sidecar_for_file(filepath) == (None, (None, None, None))
    assert db.get_previous_uuids() == []
-
    db.set_detected_text_for_uuid("FOO-BAR", json.dumps([["foo", 0.5]]))
    assert db.get_detected_text_for_uuid("FOO-BAR") is None
+    db.set_metadata_for_file(filepath, METADATA_DATA)
+    assert db.get_metadata_for_file(filepath) is None

    # test set_data which sets all at the same time
    filepath2 = os.path.join(tempdir.name, "test2.jpg")
@@ -163,6 +171,7 @@ def test_export_db_no_op():
        (10, 11, 12),
        INFO_DATA,
        EXIF_DATA,
+        METADATA_DATA,
    )
    assert db.get_uuid_for_file(filepath2) is None
    assert db.get_info_for_uuid("BAR-FOO") is None
@@ -172,6 +181,7 @@ def test_export_db_no_op():
    assert db.get_stat_converted_for_file(filepath) is None
    assert db.get_stat_edited_for_file(filepath) is None
    assert db.get_previous_uuids() == []
+    assert db.get_metadata_for_file(filepath) is None

    # update data
    db.set_uuid_for_file(filepath, "FUBAR")
@@ -207,7 +217,7 @@ def test_export_db_in_memory():
    db.set_sidecar_for_file(filepath, SIDECAR_DATA, (13, 14, 15))
    assert db.get_previous_uuids() == ["FOO-BAR"]
    db.set_detected_text_for_uuid("FOO-BAR", json.dumps([["foo", 0.5]]))
-
+    db.set_metadata_for_file(filepath, METADATA_DATA)
    db.close()

    dbram = ExportDBInMemory(dbname, tempdir.name)
@@ -226,6 +236,7 @@ def test_export_db_in_memory():
    assert dbram.get_sidecar_for_file(filepath) == (SIDECAR_DATA, (13, 14, 15))
    assert dbram.get_previous_uuids() == ["FOO-BAR"]
    assert json.loads(dbram.get_detected_text_for_uuid("FOO-BAR")) == [["foo", 0.5]]
+    assert dbram.get_metadata_for_file(filepath) == METADATA_DATA

    # change a value
    dbram.set_uuid_for_file(filepath, "FUBAR")
@@ -237,6 +248,7 @@ def test_export_db_in_memory():
    dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
    dbram.set_detected_text_for_uuid("FUBAR", json.dumps([["bar", 0.5]]))
+    dbram.set_metadata_for_file(filepath, "FUBAR")

    assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
    assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -248,6 +260,7 @@ def test_export_db_in_memory():
    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
    assert dbram.get_previous_uuids() == ["FUBAR"]
    assert json.loads(dbram.get_detected_text_for_uuid("FUBAR")) == [["bar", 0.5]]
+    assert dbram.get_metadata_for_file(filepath) == "FUBAR"

    dbram.close()

@@ -265,6 +278,7 @@ def test_export_db_in_memory():

    assert db.get_info_for_uuid("FUBAR") is None
    assert db.get_detected_text_for_uuid("FUBAR") is None
+    assert db.get_metadata_for_file(filepath) == METADATA_DATA


 def test_export_db_in_memory_nofile():
@@ -295,6 +309,7 @@ def test_export_db_in_memory_nofile():
    dbram.set_stat_edited_for_file(filepath, (4, 5, 6))
    dbram.set_sidecar_for_file(filepath, "FUBAR", (20, 21, 22))
    dbram.set_detected_text_for_uuid("FUBAR", json.dumps([["bar", 0.5]]))
+    dbram.set_metadata_for_file(filepath, METADATA_DATA)

    assert dbram.get_uuid_for_file(filepath_lower) == "FUBAR"
    assert dbram.get_info_for_uuid("FUBAR") == INFO_DATA2
@@ -306,5 +321,6 @@ def test_export_db_in_memory_nofile():
    assert dbram.get_sidecar_for_file(filepath) == ("FUBAR", (20, 21, 22))
    assert dbram.get_previous_uuids() == ["FUBAR"]
    assert json.loads(dbram.get_detected_text_for_uuid("FUBAR")) == [["bar", 0.5]]
+    assert dbram.get_metadata_for_file(filepath) == METADATA_DATA

    dbram.close()
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,27 +1,33 @@
+import logging
+import os.path
+import pathlib
+import tempfile
+
 import pytest

+import osxphotos
+
 DB_LOCKED_10_12 = "./tests/Test-Lock-10_12.photoslibrary/database/photos.db"
 DB_LOCKED_10_15 = "./tests/Test-Lock-10_15_1.photoslibrary/database/Photos.sqlite"
 DB_UNLOCKED_10_15 = "./tests/Test-10.15.1.photoslibrary/database/photos.db"

 UTI_DICT = {"public.jpeg": "jpeg", "com.canon.cr2-raw-image": "cr2"}

+from osxphotos.utils import (
+    _dd_to_dms,
+    increment_filename,
+    increment_filename_with_count,
+    list_directory,
+)
+

 def test_debug_enable():
-    import logging
-
-    import osxphotos
-
    osxphotos._set_debug(True)
    logger = osxphotos._get_logger()
    assert logger.isEnabledFor(logging.DEBUG)


 def test_debug_disable():
-    import logging
-
-    import osxphotos
-
    osxphotos._set_debug(False)
    logger = osxphotos._get_logger()
    assert not logger.isEnabledFor(logging.DEBUG)
@@ -29,14 +35,12 @@ def test_debug_disable():

 def test_dd_to_dms():
    # expands coverage for edge case in _dd_to_dms
-    from osxphotos.utils import _dd_to_dms

    assert _dd_to_dms(-0.001) == (0, 0, -3.6)


@pytest.mark.skip(reason="Fails on some machines")
 def test_get_system_library_path():
-    import osxphotos

    _, major, _ = osxphotos.utils._get_os_version()
    if int(major) < 15:
@@ -46,51 +50,73 @@ def test_get_system_library_path():


 def test_db_is_locked_locked():
-    import osxphotos

    assert osxphotos.utils._db_is_locked(DB_LOCKED_10_12)
    assert osxphotos.utils._db_is_locked(DB_LOCKED_10_15)


 def test_db_is_locked_unlocked():
-    import osxphotos

    assert not osxphotos.utils._db_is_locked(DB_UNLOCKED_10_15)


-def test_findfiles():
-    import os.path
-    import tempfile
-
-    from osxphotos.utils import findfiles
+def test_list_directory():
+    """test list_directory"""

    temp_dir = tempfile.TemporaryDirectory(prefix="osxphotos_")
-    fd = open(os.path.join(temp_dir.name, "file1.jpg"), "w+")
-    fd.close
-    fd = open(os.path.join(temp_dir.name, "file2.JPG"), "w+")
-    fd.close
-    files = findfiles("*.jpg", temp_dir.name)
+    temp_dir_name = pathlib.Path(temp_dir.name)
+    file1 = (temp_dir_name / "file1.jpg").touch()
+    file2 = (temp_dir_name / "File2.JPG").touch()
+    file3 = (temp_dir_name / "File.png").touch()
+    file4 = (temp_dir_name / "document.pdf").touch()
+
+    files = list_directory(temp_dir.name, glob="*.jpg")
    assert len(files) == 2
    assert "file1.jpg" in files
-    assert "file2.JPG" in files
+    assert "File2.JPG" in files
+    assert isinstance(files[0], str)
+
+    files = list_directory(temp_dir.name, glob="*.jpg", case_sensitive=True)
+    assert len(files) == 1
+    assert "file1.jpg" in files
+
+    files = list_directory(temp_dir.name, startswith="file")
+    assert len(files) == 3
+
+    files = list_directory(temp_dir.name, endswith="jpg")
+    assert len(files) == 2
+
+    files = list_directory(temp_dir.name, contains="doc")
+    assert len(files) == 1
+    assert "document.pdf" in files
+
+    files = list_directory(temp_dir.name, startswith="File", case_sensitive=True)
+    assert len(files) == 2
+
+    files = list_directory(temp_dir.name, startswith="File", case_sensitive=False)
+    assert len(files) == 3
+
+    files = list_directory(temp_dir.name, startswith="document", include_path=True)
+    assert len(files) == 1
+    assert files[0] == str(pathlib.Path(temp_dir.name) / "document.pdf")
+
+    # test pathlib.Path
+    files = list_directory(temp_dir_name, glob="*.jpg")
+    assert isinstance(files[0], pathlib.Path)
+
+    files = list_directory(temp_dir.name, glob="FooBar*.jpg")
+    assert not files


-def test_findfiles_invalid_dir():
-    import tempfile
-
-    from osxphotos.utils import findfiles
+def test_list_directory_invalid():

    temp_dir = tempfile.TemporaryDirectory(prefix="osxphotos_")
-    files = findfiles("*.jpg", f"{temp_dir.name}/no_such_dir")
+    files = list_directory(f"{temp_dir.name}/no_such_dir", glob="*.jpg")
    assert len(files) == 0


 def test_increment_filename():
    # test that increment_filename works
-    import pathlib
-    import tempfile
-
-    from osxphotos.utils import increment_filename, increment_filename_with_count

    with tempfile.TemporaryDirectory(prefix="osxphotos_") as temp_dir:
        temp_dir = pathlib.Path(temp_dir)
--- a/utils/exiftool_supported_types.py
+++ b/utils/exiftool_supported_types.py
@@ -0,0 +1,57 @@
+"""Read the "Supported File Types" table from exiftool.org and build a json file from the table"""
+
+import json
+import sys
+
+import requests
+from bs4 import BeautifulSoup
+
+
+if __name__ == "__main__":
+    url = "https://www.exiftool.org/"
+    json_file = "exiftool_filetypes.json"
+
+    html_content = requests.get(url).text
+
+    soup = BeautifulSoup(html_content, "html.parser")
+
+    # uncomment to see all table classes
+    # print("Classes of each table:")
+    # for table in soup.find_all("table"):
+    #     print(table.get("class"))
+
+    # strip footnotes in <span> tags
+    for span_tag in soup.findAll("span"):
+        span_tag.replace_with("")
+
+    # find the table for Supported File Types
+    table = soup.find("table", class_="sticky tight sm bm")
+
+    # get table headers
+    table_headers = [tx.text.lower() for tx in table.find_all("th")]
+
+    # get table data
+    table_data = []
+    for tr in table.find_all("tr"):
+        if row := [td.text for td in tr.find_all("td")]:
+            table_data.append(row)
+
+    # make a dictionary of the table data
+    supported_filetypes = {}
+    for row in table_data:
+        row_dict = dict(zip(table_headers, row))
+        for key, value in row_dict.items():
+            if value == "-":
+                row_dict[key] = None
+        row_dict["file type"] = row_dict["file type"].split(",")
+        row_dict["file type"] = [ft.strip() for ft in row_dict["file type"]]
+        row_dict["read"] = "R" in row_dict["support"]
+        row_dict["write"] = "W" in row_dict["support"]
+        row_dict["create"] = "C" in row_dict["support"]
+        filetypes = [ft.lower() for ft in row_dict["file type"]]
+        for filetype in filetypes:
+            supported_filetypes[filetype] = {"extension": filetype, **row_dict}
+
+    with open(json_file, "w") as jsonfile:
+        print(f"Writing {json_file}...")
+        json.dump(supported_filetypes, jsonfile, indent=4)
Author	SHA1	Message	Date
Rhet Turnbull	30abdddaf3	Added --force-update, #621	2022-02-12 21:01:16 -08:00
Rhet Turnbull	a2f329b8de	Updated CHANGELOG.md [skip ci]	2022-02-12 17:59:48 -08:00
Rhet Turnbull	bfa888adc5	Added --force-update, #621	2022-02-12 17:49:40 -08:00
Rhet Turnbull	ac4083bfbb	Fix for #630	2022-02-12 00:23:50 -08:00
Rhet Turnbull	5fb686ac0c	Refactored fix for #627	2022-02-11 23:10:09 -08:00
Rhet Turnbull	49a7b80680	Fixed cleanup for #629	2022-02-11 06:18:17 -08:00
Rhet Turnbull	cb11967eac	Implement #629 , sqlite performance optimizatons for export db	2022-02-10 22:36:35 -08:00
Rhet Turnbull	a43bfc5a33	Updated CHANGELOG.md [skip ci]	2022-02-06 00:01:43 -08:00
Rhet Turnbull	1d6bc4e09e	Additional fix for #615	2022-02-05 23:57:50 -08:00
Rhet Turnbull	3e14b718ef	Updated docs [skip ci]	2022-02-05 23:12:42 -08:00
Rhet Turnbull	1ae6270561	Fixed exiftool to ignore unsupported file types, #615	2022-02-05 22:54:50 -08:00
Rhet Turnbull	55a601c07e	Updated tests	2022-02-05 14:30:20 -08:00
Rhet Turnbull	7d67b81879	Updated CHANGELOG.md [skip ci]	2022-02-05 14:08:43 -08:00
Rhet Turnbull	cd02144ac3	Fix for --name searching only original_filename on Photos 5+, #594	2022-02-05 12:55:56 -08:00
Rhet Turnbull	9b247acd1c	Fix for unicode in query strings, #618	2022-02-05 12:36:25 -08:00
Rhet Turnbull	942126ea3d	Updated CHANGELOG.md [skip ci]	2022-02-05 10:56:18 -08:00
Rhet Turnbull	2b9ea11701	Updated docs [skip ci]	2022-02-05 10:39:35 -08:00
Rhet Turnbull	b3d3e14ffe	Fix for #561 , no really, I mean it this time	2022-02-05 10:36:23 -08:00
Rhet Turnbull	62ae5db9fd	Updated CHANGELOG.md [skip ci]	2022-02-04 21:59:33 -08:00
Rhet Turnbull	77a49a09a1	Updated tests for #561 [skip ci]	2022-02-04 05:56:01 -08:00
Rhet Turnbull	06c5bbfcfd	Updated docs [skip ci]	2022-02-03 22:49:56 -08:00
Rhet Turnbull	f3063d35be	Fix for filenames with special characters, #561 , #618	2022-02-03 22:46:11 -08:00
Rhet Turnbull	e32090bf39	Updated known issues [skip ci]	2022-02-01 06:53:25 -08:00
Rhet Turnbull	7ab500740b	Added progress counter, #601	2022-01-29 19:02:25 -08:00
Rhet Turnbull	911bd30d28	Updated CHANGELOG.md [skip ci]	2022-01-29 19:02:00 -08:00
allcontributors[bot]	282857eae0	docs: add oPromessa as a contributor for ideas, test (#611 ) * docs: update .all-contributorsrc [skip ci] * docs: update README.md [skip ci] Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>	2022-01-29 14:08:36 -08:00
Rhet Turnbull	d8c2f99c06	Added --timestamp option for --verbose, #600	2022-01-29 11:59:41 -08:00
Rhet Turnbull	16d3f74366	Updated formatting for elapsed time, #604	2022-01-29 11:05:33 -08:00
Rhet Turnbull	5fc28139ea	Updated docs [skip ci]	2022-01-29 10:55:41 -08:00
Rhet Turnbull	b7b6876688	Updated CHANGELOG.md [skip ci]	2022-01-29 10:03:31 -08:00