Fix for --name searching only original_filename on Photos 5+, #594

* docs: update .all-contributorsrc [skip ci]

* docs: update README.md [skip ci]

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

.all-contributorsrc

@@ -257,7 +257,9 @@
       "avatar_url": "https://avatars.githubusercontent.com/u/21261491?v=4",
       "profile": "https://github.com/oPromessa",
       "contributions": [
-        "bug"
+        "bug",
+        "ideas",
+        "test"
       ]
     },
     {

CHANGELOG.md

@@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file. Dates are d
 
 Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
 
+#### [v0.45.5](https://github.com/RhetTbull/osxphotos/compare/v0.45.4...v0.45.5)
+
+> 5 February 2022
+
+- Fix for #561, no really, I mean it this time [`b3d3e14`](https://github.com/RhetTbull/osxphotos/commit/b3d3e14ffe41fbb22edb614b24f3985f379766a2)
+- Updated docs [skip ci] [`2b9ea11`](https://github.com/RhetTbull/osxphotos/commit/2b9ea11701799af9a661a8e2af70fca97235f487)
+- Updated tests for #561 [skip ci] [`77a49a0`](https://github.com/RhetTbull/osxphotos/commit/77a49a09a1bee74113a7114c543fbc25fa410ffc)
+
+#### [v0.45.4](https://github.com/RhetTbull/osxphotos/compare/v0.45.3...v0.45.4)
+
+> 3 February 2022
+
+- docs: add oPromessa as a contributor for ideas, test [`#611`](https://github.com/RhetTbull/osxphotos/pull/611)
+- Fix for filenames with special characters, #561, #618 [`f3063d3`](https://github.com/RhetTbull/osxphotos/commit/f3063d35be3c96342d83dbd87ddd614a2001bff4)
+- Updated docs [skip ci] [`06c5bbf`](https://github.com/RhetTbull/osxphotos/commit/06c5bbfcfdf591a4a5d43f1456adaa27385fe01a)
+- Added progress counter, #601 [`7ab5007`](https://github.com/RhetTbull/osxphotos/commit/7ab500740b28594dcd778140e10991f839220e9d)
+- Updated known issues [skip ci] [`e32090b`](https://github.com/RhetTbull/osxphotos/commit/e32090bf39cb786171b49443f878ffdbab774420)
+
+#### [v0.45.3](https://github.com/RhetTbull/osxphotos/compare/v0.45.2...v0.45.3)
+
+> 29 January 2022
+
+- Added --timestamp option for --verbose, #600 [`d8c2f99`](https://github.com/RhetTbull/osxphotos/commit/d8c2f99c06bc6f72bf2cb1a13c5765824fe3cbba)
+- Updated docs [skip ci] [`5fc2813`](https://github.com/RhetTbull/osxphotos/commit/5fc28139ea0374bc3e228c0432b8a41ada430389)
+- Updated formatting for elapsed time, #604 [`16d3f74`](https://github.com/RhetTbull/osxphotos/commit/16d3f743664396d43b3b3028a5e7a919ec56d9e1)
+
 #### [v0.45.2](https://github.com/RhetTbull/osxphotos/compare/v0.45.0...v0.45.2)
 
 > 29 January 2022

README.md

@@ -1725,7 +1725,7 @@ Substitution                    Description
 {lf}                            A line feed: '\n', alias for {newline}
 {cr}                            A carriage return: '\r'
 {crlf}                          a carriage return + line feed: '\r\n'
-{osxphotos_version}             The osxphotos version, e.g. '0.45.3'
+{osxphotos_version}             The osxphotos version, e.g. '0.45.6'
 {osxphotos_cmd_line}            The full command line used to run osxphotos
 
 The following substitutions may result in multiple values. Thus if specified for
@@ -3629,7 +3629,7 @@ The following template field substitutions are availabe for use the templating s
 |{lf}|A line feed: '\n', alias for {newline}|
 |{cr}|A carriage return: '\r'|
 |{crlf}|a carriage return + line feed: '\r\n'|
-|{osxphotos_version}|The osxphotos version, e.g. '0.45.3'|
+|{osxphotos_version}|The osxphotos version, e.g. '0.45.6'|
 |{osxphotos_cmd_line}|The full command line used to run osxphotos|
 |{album}|Album(s) photo is contained in|
 |{folder_album}|Folder path + album photo is contained in. e.g. 'Folder/Subfolder/Album' or just 'Album' if no enclosing folder|
@@ -3948,7 +3948,7 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
     <td align="center"><a href="https://github.com/mkirkland4874"><img src="https://avatars.githubusercontent.com/u/36466711?v=4?s=75" width="75px;" alt=""/><br /><sub><b>mkirkland4874</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Amkirkland4874" title="Bug reports">🐛</a> <a href="#example-mkirkland4874" title="Examples">💡</a></td>
     <td align="center"><a href="https://github.com/jcommisso07"><img src="https://avatars.githubusercontent.com/u/3111054?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Joseph Commisso</b></sub></a><br /><a href="#data-jcommisso07" title="Data">🔣</a></td>
     <td align="center"><a href="https://github.com/dssinger"><img src="https://avatars.githubusercontent.com/u/1817903?v=4?s=75" width="75px;" alt=""/><br /><sub><b>David Singer</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Adssinger" title="Bug reports">🐛</a></td>
-    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a></td>
+    <td align="center"><a href="https://github.com/oPromessa"><img src="https://avatars.githubusercontent.com/u/21261491?v=4?s=75" width="75px;" alt=""/><br /><sub><b>oPromessa</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3AoPromessa" title="Bug reports">🐛</a> <a href="#ideas-oPromessa" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/RhetTbull/osxphotos/commits?author=oPromessa" title="Tests">⚠️</a></td>
     <td align="center"><a href="http://spencerchang.me"><img src="https://avatars.githubusercontent.com/u/14796580?v=4?s=75" width="75px;" alt=""/><br /><sub><b>Spencer Chang</b></sub></a><br /><a href="https://github.com/RhetTbull/osxphotos/issues?q=author%3Aspencerc99" title="Bug reports">🐛</a></td>
   </tr>
   <tr>
@@ -3974,7 +3974,6 @@ My goal is make osxphotos as reliable and comprehensive as possible.  The test s
 
 - Audio-only files are not handled.  It is possible to store audio-only files in Photos.  osxphotos currently only handles images and videos. See [Issue #436](https://github.com/RhetTbull/osxphotos/issues/436)
 - Face coordinates (mouth, left eye, right eye) may not be correct for images where the head is tilted.  See [Issue #196](https://github.com/RhetTbull/osxphotos/issues/196).
-- Raw images imported to Photos with an associated jpeg preview are not handled correctly by osxphotos.  osxphotos query and export will operate on the jpeg preview instead of the raw image as will `PhotoInfo.path`.  If the user selects "Use RAW as original" in Photos, the raw image will be exported or operated on but the jpeg will be ignored.  See [Issue #101](https://github.com/RhetTbull/osxphotos/issues/101). Note: Beta version of fix for this bug is implemented in the current version of osxphotos.
 - The `--download-missing` option for `osxphotos export` does not work correctly with burst images.  It will download the primary image but not the other burst images.  See [Issue #75](https://github.com/RhetTbull/osxphotos/issues/75).
 
 ## Implementation Notes

docs/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: cf6ba426eed8ae7fdcc87f4fab23a946
+config: 3c4bdd115410fda411407689f33d7c4c
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/_static/documentation_options.js

@@ -1,6 +1,6 @@
 var DOCUMENTATION_OPTIONS = {
     URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
-    VERSION: '0.45.3',
+    VERSION: '0.45.6',
     LANGUAGE: 'None',
     COLLAPSE_INDEX: false,
     BUILDER: 'html',

docs/cli.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.45.3 documentation</title>
+    <title>osxphotos command line interface (CLI) &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/genindex.html

@@ -5,7 +5,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Index &#8212; osxphotos 0.45.3 documentation</title>
+    <title>Index &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/index.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.45.3 documentation</title>
+    <title>Welcome to osxphotos’s documentation! &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/modules.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos &#8212; osxphotos 0.45.3 documentation</title>
+    <title>osxphotos &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/reference.html

@@ -6,7 +6,7 @@
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
 
-    <title>osxphotos package &#8212; osxphotos 0.45.3 documentation</title>
+    <title>osxphotos package &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>

docs/search.html

@@ -5,7 +5,7 @@
   <head>
     <meta charset="utf-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>Search &#8212; osxphotos 0.45.3 documentation</title>
+    <title>Search &#8212; osxphotos 0.45.6 documentation</title>
     <link rel="stylesheet" type="text/css" href="_static/pygments.css" />
     <link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
     

osxphotos/_version.py

@@ -1,3 +1,3 @@
 """ version info """
 
-__version__ = "0.45.3"
+__version__ = "0.45.6"

osxphotos/cli.py

@@ -1896,10 +1896,12 @@ def export(
             else None
         )
 
+        photo_num = 0
         # send progress bar output to /dev/null if verbose to hide the progress bar
         fp = open(os.devnull, "w") if verbose else None
-        with click.progressbar(photos, file=fp) as bar:
+        with click.progressbar(photos, show_pos=True, file=fp) as bar:
             for p in bar:
+                photo_num += 1
                 export_results = export_photo(
                     photo=p,
                     dest=dest,
@@ -1946,6 +1948,8 @@ def export(
                     export_preview=preview,
                     preview_suffix=preview_suffix,
                     preview_if_missing=preview_if_missing,
+                    photo_num=photo_num,
+                    num_photos=num_photos,
                 )
 
                 if post_function:
@@ -2624,6 +2628,8 @@ def export_photo(
     export_preview=False,
     preview_suffix=None,
     preview_if_missing=False,
+    photo_num=1,
+    num_photos=1,
 ):
     """Helper function for export that does the actual export
 
@@ -2668,6 +2674,8 @@ def export_photo(
         export_preview: export the preview image generated by Photos
         preview_suffix: str, template to use as suffix for preview images
         preview_if_missing: bool, export preview if original is missing
+        photo_num: int, which number photo in total of num_photos is being exported
+        num_photos: int, total number of photos that will be exported
 
     Returns:
         list of path(s) of exported photo or None if photo was missing
@@ -2789,7 +2797,7 @@ def export_photo(
             original_filename = str(original_filename)
 
             verbose_(
-                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename}"
+                f"Exporting {photo.original_filename} ({photo.filename}) as {original_filename} ({photo_num}/{num_photos})"
             )
 
             results += export_photo_to_directory(

osxphotos/export_db.py

@@ -10,13 +10,17 @@ import sys
 from abc import ABC, abstractmethod
 from io import StringIO
 from sqlite3 import Error
+from typing import Union
 
 from ._constants import OSXPHOTOS_EXPORT_DB
 from ._version import __version__
+from .utils import normalize_fs_path
 
 __all__ = ["ExportDB_ABC", "ExportDBNoOp", "ExportDB", "ExportDBInMemory"]
 
-OSXPHOTOS_EXPORTDB_VERSION = "4.2"
+OSXPHOTOS_EXPORTDB_VERSION = "4.3"
+OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH = "4.3"
+
 OSXPHOTOS_ABOUT_STRING = f"Created by osxphotos version {__version__} (https://github.com/RhetTbull/osxphotos) on {datetime.datetime.now()}"
 
 
@@ -211,12 +215,13 @@ class ExportDB(ExportDB_ABC):
         """query database for filename and return UUID
         returns None if filename not found in database
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filepath_normalized = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
             c.execute(
-                "SELECT uuid FROM files WHERE filepath_normalized = ?", (filename,)
+                "SELECT uuid FROM files WHERE filepath_normalized = ?",
+                (filepath_normalized,),
             )
             results = c.fetchone()
             uuid = results[0] if results else None
@@ -228,7 +233,7 @@ class ExportDB(ExportDB_ABC):
     def set_uuid_for_file(self, filename, uuid):
         """set UUID of filename to uuid in the database"""
         filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -245,7 +250,7 @@ class ExportDB(ExportDB_ABC):
         """set stat info for filename
         filename: filename to set the stat info for
         stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -266,7 +271,7 @@ class ExportDB(ExportDB_ABC):
         """get stat info for filename
         returns: tuple of (mode, size, mtime)
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -302,7 +307,7 @@ class ExportDB(ExportDB_ABC):
         """set stat info for filename (after exiftool has updated it)
         filename: filename to set the stat info for
         stat: a tuple of length 3: mode, size, mtime"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -323,7 +328,7 @@ class ExportDB(ExportDB_ABC):
         """get stat info for filename (after exiftool has updated it)
         returns: tuple of (mode, size, mtime)
         """
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -384,7 +389,7 @@ class ExportDB(ExportDB_ABC):
 
     def get_exifdata_for_file(self, filename):
         """returns the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -402,7 +407,7 @@ class ExportDB(ExportDB_ABC):
 
     def set_exifdata_for_file(self, filename, exifdata):
         """sets the exifdata JSON struct for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -416,7 +421,7 @@ class ExportDB(ExportDB_ABC):
 
     def get_sidecar_for_file(self, filename):
         """returns the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -444,7 +449,7 @@ class ExportDB(ExportDB_ABC):
 
     def set_sidecar_for_file(self, filename, sidecar_data, sidecar_sig):
         """sets the sidecar data and signature for a file"""
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -515,7 +520,7 @@ class ExportDB(ExportDB_ABC):
     ):
         """sets all the data for file and uuid at once; if any value is None, does not set it"""
         filename = str(pathlib.Path(filename).relative_to(self._path))
-        filename_normalized = filename.lower()
+        filename_normalized = self._normalize_filepath(filename)
         conn = self._conn
         try:
             c = conn.cursor()
@@ -577,7 +582,7 @@ class ExportDB(ExportDB_ABC):
             logging.warning(e)
 
     def _set_stat_for_file(self, table, filename, stats):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         if len(stats) != 3:
             raise ValueError(f"expected 3 elements for stat, got {len(stats)}")
 
@@ -590,7 +595,7 @@ class ExportDB(ExportDB_ABC):
         conn.commit()
 
     def _get_stat_for_file(self, table, filename):
-        filename = str(pathlib.Path(filename).relative_to(self._path)).lower()
+        filename = self._normalize_filepath_relative(filename)
         conn = self._conn
         c = conn.cursor()
         c.execute(
@@ -626,6 +631,8 @@ class ExportDB(ExportDB_ABC):
             version_info = self._get_database_version(conn)
             if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION:
                 self._create_db_tables(conn)
+                if version_info[1] < OSXPHOTOS_EXPORTDB_VERSION_MIGRATE_FILEPATH:
+                    self._migrate_normalized_filepath(conn)
                 self.was_upgraded = (version_info[1], OSXPHOTOS_EXPORTDB_VERSION)
             else:
                 self.was_upgraded = ()
@@ -782,6 +789,32 @@ class ExportDB(ExportDB_ABC):
         except Error as e:
             logging.warning(e)
 
+    def _normalize_filepath(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, lower case"""
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _normalize_filepath_relative(self, filepath: Union[str, pathlib.Path]) -> str:
+        """normalize filepath for unicode, relative path (to export dir), lower case"""
+        filepath = str(pathlib.Path(filepath).relative_to(self._path))
+        return normalize_fs_path(str(filepath)).lower()
+
+    def _migrate_normalized_filepath(self, conn):
+        """Fix all filepath_normalized columns for unicode normalization"""
+        # Prior to database version 4.3, filepath_normalized was not normalized for unicode
+        c = conn.cursor()
+        for table in ["converted", "edited", "exifdata", "files", "sidecar"]:
+            old_values = c.execute(
+                f"SELECT filepath_normalized, id FROM {table}"
+            ).fetchall()
+            new_values = [
+                (self._normalize_filepath(filepath_normalized), id_)
+                for filepath_normalized, id_ in old_values
+            ]
+            c.executemany(
+                f"UPDATE {table} SET filepath_normalized=? WHERE id=?", new_values
+            )
+        conn.commit()
+
 
 class ExportDBInMemory(ExportDB):
     """In memory version of ExportDB

osxphotos/photoexporter.py

@@ -45,7 +45,7 @@ from .photokit import (
 )
 from .phototemplate import RenderOptions
 from .uti import get_preferred_uti_extension
-from .utils import increment_filename, increment_filename_with_count, lineno
+from .utils import increment_filename, lineno, list_directory
 
 __all__ = [
     "ExportError",
@@ -598,9 +598,13 @@ class PhotoExporter:
                 )
             if dest_uuid != self.photo.uuid:
                 # not the right file, find the right one
-                glob_str = str(dest.parent / f"{dest.stem} (*{dest.suffix}")
-                # TODO: use the normalized code in utils
-                dest_files = glob.glob(glob_str)
+                # find files that match "dest_name (*.ext" (e.g. "dest_name (1).jpg", "dest_name (2).jpg)", ...)
+                dest_files = list_directory(
+                    dest.parent,
+                    startswith=f"{dest.stem} (",
+                    endswith=dest.suffix,
+                    include_path=True,
+                )
                 for file_ in dest_files:
                     dest_uuid = export_db.get_uuid_for_file(file_)
                     if dest_uuid == self.photo.uuid:
@@ -1828,7 +1832,7 @@ def _export_photo_uuid_applescript(
         raise ValueError(f"dest {dest} must be a directory")
 
     if not original ^ edited:
-        raise ValueError(f"edited or original must be True but not both")
+        raise ValueError("edited or original must be True but not both")
 
     tmpdir = tempfile.TemporaryDirectory(prefix="osxphotos_")
 
@@ -1851,7 +1855,6 @@ def _export_photo_uuid_applescript(
     if not exported_files or not filename:
         # nothing got exported
         raise ExportError(f"Could not export photo {uuid} ({lineno(__file__)})")
-
     # need to find actual filename as sometimes Photos renames JPG to jpeg on export
     # may be more than one file exported (e.g. if Live Photo, Photos exports both .jpeg and .mov)
     # TemporaryDirectory will cleanup on return

osxphotos/photoinfo.py

@@ -54,7 +54,7 @@ from .scoreinfo import ScoreInfo
 from .searchinfo import SearchInfo
 from .text_detection import detect_text
 from .uti import get_preferred_uti_extension, get_uti_for_extension
-from .utils import _debug, _get_resource_loc, findfiles
+from .utils import _debug, _get_resource_loc, list_directory
 
 __all__ = ["PhotoInfo", "PhotoInfoNone"]
 
@@ -369,7 +369,7 @@ class PhotoInfo:
         # In Photos 5, raw is in same folder as original but with _4.ext
         # Unless "Copy Items to the Photos Library" is not checked
         # then RAW image is not renamed but has same name is jpeg buth with raw extension
-        # Current implementation uses findfiles to find images with the correct raw UTI extension
+        # Current implementation finds images with the correct raw UTI extension
         # in same folder as the original and with same stem as original in form: original_stem*.raw_ext
         # TODO: I don't like this -- would prefer a more deterministic approach but until I have more
         # data on how Photos stores and retrieves RAW images, this seems to be working
@@ -405,8 +405,7 @@ class PhotoInfo:
             # raw files have same name as original but with _4.raw_ext appended
             # I believe the _4 maps to PHAssetResourceTypeAlternatePhoto = 4
             # see: https://developer.apple.com/documentation/photokit/phassetresourcetype/phassetresourcetypealternatephoto?language=objc
-            glob_str = f"{filestem}_4*"
-            raw_file = findfiles(glob_str, filepath)
+            raw_file = list_directory(filepath, startswith=f"{filestem}_4")
             if not raw_file:
                 photopath = None
             else:

osxphotos/photosdb/photosdb.py

@@ -39,6 +39,7 @@ from .._constants import (
     _PHOTOS_5_PROJECT_ALBUM_KIND,
     _PHOTOS_5_ROOT_FOLDER_KIND,
     _PHOTOS_5_SHARED_ALBUM_KIND,
+    _PHOTOS_5_VERSION,
     _TESTED_OS_VERSIONS,
     _UNKNOWN_PERSON,
     BURST_KEY,
@@ -659,14 +660,18 @@ class PhotosDB:
 
         for person in c:
             pk = person[0]
-            fullname = person[2] if person[2] is not None else _UNKNOWN_PERSON
+            fullname = (
+                normalize_unicode(person[2])
+                if person[2] is not None
+                else _UNKNOWN_PERSON
+            )
             self._dbpersons_pk[pk] = {
                 "pk": pk,
                 "uuid": person[1],
                 "fullname": fullname,
                 "facecount": person[3],
                 "keyface": person[5],
-                "displayname": person[4],
+                "displayname": normalize_unicode(person[4]),
                 "photo_uuid": None,
                 "keyface_uuid": None,
             }
@@ -733,13 +738,6 @@ class PhotosDB:
             except KeyError:
                 self._dbfaces_pk[pk] = [uuid]
 
-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
         # Get info on albums
         verbose("Processing albums.")
         c.execute(
@@ -876,14 +874,6 @@ class PhotosDB:
             else:
                 self._dbalbum_folders[album] = {}
 
-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-            logging.debug(pformat(self._dbfolder_details))
-
         # Get info on keywords
         verbose("Processing keywords.")
         c.execute(
@@ -899,13 +889,16 @@ class PhotosDB:
                 RKMaster.uuid = RKVersion.masterUuid
             """
         )
-        for keyword in c:
-            if not keyword[1] in self._dbkeywords_uuid:
-                self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword[0] in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword[0]] = []
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword[0]].append(keyword[1])
+        for keyword_title, keyword_uuid, _ in c:
+            keyword_title = normalize_unicode(keyword_title)
+            try:
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
+            except KeyError:
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
+            try:
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
+            except KeyError:
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]
 
         # Get info on disk volumes
         c.execute("select RKVolume.modelId, RKVolume.name from RKVolume")
@@ -1027,13 +1020,11 @@ class PhotosDB:
 
         for row in c:
             uuid = row[0]
-            if _debug():
-                logging.debug(f"uuid = '{uuid}, master = '{row[2]}")
             self._dbphotos[uuid] = {}
             self._dbphotos[uuid]["_uuid"] = uuid  # stored here for easier debugging
             self._dbphotos[uuid]["modelID"] = row[1]
             self._dbphotos[uuid]["masterUuid"] = row[2]
-            self._dbphotos[uuid]["filename"] = row[3]
+            self._dbphotos[uuid]["filename"] = normalize_unicode(row[3])
 
             # There are sometimes negative values for lastmodifieddate in the database
             # I don't know what these mean but they will raise exception in datetime if
@@ -1272,13 +1263,13 @@ class PhotosDB:
             info["volumeId"] = row[1]
             info["imagePath"] = row[2]
             info["isMissing"] = row[3]
-            info["originalFilename"] = row[4]
+            info["originalFilename"] = normalize_unicode(row[4])
             info["UTI"] = row[5]
             info["modelID"] = row[6]
             info["fileSize"] = row[7]
             info["isTrulyRAW"] = row[8]
             info["alternateMasterUuid"] = row[9]
-            info["filename"] = row[10]
+            info["filename"] = normalize_unicode(row[10])
             self._dbphotos_master[uuid] = info
 
         # get details needed to find path of the edited photos
@@ -1550,39 +1541,6 @@ class PhotosDB:
 
         # done processing, dump debug data if requested
         verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))
 
     def _build_album_folder_hierarchy_4(self, uuid, folders=None):
         """recursively build folder/album hierarchy
@@ -1673,7 +1631,7 @@ class PhotosDB:
         for person in c:
             pk = person[0]
             fullname = (
-                person[2]
+                normalize_unicode(person[2])
                 if (person[2] != "" and person[2] is not None)
                 else _UNKNOWN_PERSON
             )
@@ -1683,7 +1641,7 @@ class PhotosDB:
                 "fullname": fullname,
                 "facecount": person[3],
                 "keyface": person[4],
-                "displayname": person[5],
+                "displayname": normalize_unicode(person[5]),
                 "photo_uuid": None,
                 "keyface_uuid": None,
             }
@@ -1747,13 +1705,6 @@ class PhotosDB:
             except KeyError:
                 self._dbfaces_pk[pk] = [uuid]
 
-        if _debug():
-            logging.debug(f"Finished walking through persons")
-            logging.debug(pformat(self._dbpersons_pk))
-            logging.debug(pformat(self._dbpersons_fullname))
-            logging.debug(pformat(self._dbfaces_pk))
-            logging.debug(pformat(self._dbfaces_uuid))
-
         # get details about albums
         verbose("Processing albums.")
         c.execute(
@@ -1870,13 +1821,6 @@ class PhotosDB:
                 # shared albums can't be in folders
                 self._dbalbum_folders[album] = []
 
-        if _debug():
-            logging.debug(f"Finished walking through albums")
-            logging.debug(pformat(self._dbalbums_album))
-            logging.debug(pformat(self._dbalbums_uuid))
-            logging.debug(pformat(self._dbalbum_details))
-            logging.debug(pformat(self._dbalbum_folders))
-
         # get details on keywords
         verbose("Processing keywords.")
         c.execute(
@@ -1886,29 +1830,22 @@ class PhotosDB:
                 JOIN Z_1KEYWORDS ON Z_1KEYWORDS.Z_1ASSETATTRIBUTES = ZADDITIONALASSETATTRIBUTES.Z_PK 
                 JOIN ZKEYWORD ON ZKEYWORD.Z_PK = {keyword_join} """
         )
-        for keyword in c:
-            keyword_title = normalize_unicode(keyword[0])
-            if not keyword[1] in self._dbkeywords_uuid:
-                self._dbkeywords_uuid[keyword[1]] = []
-            if not keyword_title in self._dbkeywords_keyword:
-                self._dbkeywords_keyword[keyword_title] = []
-            self._dbkeywords_uuid[keyword[1]].append(keyword[0])
-            self._dbkeywords_keyword[keyword_title].append(keyword[1])
-
-        if _debug():
-            logging.debug(f"Finished walking through keywords")
-            logging.debug(pformat(self._dbkeywords_keyword))
-            logging.debug(pformat(self._dbkeywords_uuid))
+        for keyword_title, keyword_uuid in c:
+            keyword_title = normalize_unicode(keyword_title)
+            try:
+                self._dbkeywords_uuid[keyword_uuid].append(keyword_title)
+            except KeyError:
+                self._dbkeywords_uuid[keyword_uuid] = [keyword_title]
+            try:
+                self._dbkeywords_keyword[keyword_title].append(keyword_uuid)
+            except KeyError:
+                self._dbkeywords_keyword[keyword_title] = [keyword_uuid]
 
         # get details on disk volumes
         c.execute("SELECT ZUUID, ZNAME from ZFILESYSTEMVOLUME")
         for vol in c:
             self._dbvolumes[vol[0]] = vol[1]
 
-        if _debug():
-            logging.debug(f"Finished walking through volumes")
-            logging.debug(self._dbvolumes)
-
         # get details about photos
         verbose("Processing photo details.")
         c.execute(
@@ -2042,8 +1979,8 @@ class PhotosDB:
 
             info["hidden"] = row[9]
             info["favorite"] = row[10]
-            info["originalFilename"] = row[3]
-            info["filename"] = row[12]
+            info["originalFilename"] = normalize_unicode(row[3])
+            info["filename"] = normalize_unicode(row[12])
             info["directory"] = row[11]
 
             # set latitude and longitude
@@ -2521,48 +2458,6 @@ class PhotosDB:
 
         # done processing, dump debug data if requested
         verbose("Done processing details from Photos library.")
-        if _debug():
-            logging.debug("Faces (_dbfaces_uuid):")
-            logging.debug(pformat(self._dbfaces_uuid))
-
-            logging.debug("Persons (_dbpersons_pk):")
-            logging.debug(pformat(self._dbpersons_pk))
-
-            logging.debug("Keywords by uuid (_dbkeywords_uuid):")
-            logging.debug(pformat(self._dbkeywords_uuid))
-
-            logging.debug("Keywords by keyword (_dbkeywords_keywords):")
-            logging.debug(pformat(self._dbkeywords_keyword))
-
-            logging.debug("Albums by uuid (_dbalbums_uuid):")
-            logging.debug(pformat(self._dbalbums_uuid))
-
-            logging.debug("Albums by album (_dbalbums_albums):")
-            logging.debug(pformat(self._dbalbums_album))
-
-            logging.debug("Album details (_dbalbum_details):")
-            logging.debug(pformat(self._dbalbum_details))
-
-            logging.debug("Album titles (_dbalbum_titles):")
-            logging.debug(pformat(self._dbalbum_titles))
-
-            logging.debug("Album folders (_dbalbum_folders):")
-            logging.debug(pformat(self._dbalbum_folders))
-
-            logging.debug("Album parent folders (_dbalbum_parent_folders):")
-            logging.debug(pformat(self._dbalbum_parent_folders))
-
-            logging.debug("Albums pk (_dbalbums_pk):")
-            logging.debug(pformat(self._dbalbums_pk))
-
-            logging.debug("Volumes (_dbvolumes):")
-            logging.debug(pformat(self._dbvolumes))
-
-            logging.debug("Photos (_dbphotos):")
-            logging.debug(pformat(self._dbphotos))
-
-            logging.debug("Burst Photos (dbphotos_burst:")
-            logging.debug(pformat(self._dbphotos_burst))
 
     def _process_moments(self):
         """Process data from ZMOMENT table"""
@@ -2623,8 +2518,8 @@ class PhotosDB:
             moment_info["modificationDate"] = row[6]
             moment_info["representativeDate"] = row[7]
             moment_info["startDate"] = row[8]
-            moment_info["subtitle"] = row[9]
-            moment_info["title"] = row[10]
+            moment_info["subtitle"] = normalize_unicode(row[9])
+            moment_info["title"] = normalize_unicode(row[10])
             moment_info["uuid"] = row[11]
 
             # if both lat/lon == -180, then it means location undefined
@@ -3027,6 +2922,7 @@ class PhotosDB:
             if keywords:
                 keyword_set = set()
                 for keyword in keywords:
+                    keyword = normalize_unicode(keyword)
                     if keyword in self._dbkeywords_keyword:
                         keyword_set.update(self._dbkeywords_keyword[keyword])
                 photos_sets.append(keyword_set)
@@ -3034,6 +2930,7 @@ class PhotosDB:
             if persons:
                 person_set = set()
                 for person in persons:
+                    person = normalize_unicode(person)
                     if person in self._dbpersons_fullname:
                         for pk in self._dbpersons_fullname[person]:
                             try:
@@ -3076,8 +2973,6 @@ class PhotosDB:
                 ):
                     info = PhotoInfo(db=self, uuid=p, info=self._dbphotos[p])
                     photoinfo.append(info)
-        if _debug:
-            logging.debug(f"photoinfo: {pformat(photoinfo)}")
 
         return photoinfo
 
@@ -3414,23 +3309,35 @@ class PhotosDB:
                 # case-insensitive
                 for n in name:
                     n = n.lower()
-                    photo_list.extend(
-                        [
-                            p
-                            for p in photos
-                            if n in p.filename.lower()
-                            or n in p.original_filename.lower()
-                        ]
-                    )
+                    if self._db_version >= _PHOTOS_5_VERSION:
+                        # search only original_filename (#594)
+                        photo_list.extend(
+                            [p for p in photos if n in p.original_filename.lower()]
+                        )
+                    else:
+                        photo_list.extend(
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename.lower()
+                                or n in p.original_filename.lower()
+                            ]
+                        )
             else:
                 for n in name:
-                    photo_list.extend(
-                        [
-                            p
-                            for p in photos
-                            if n in p.filename or n in p.original_filename
-                        ]
-                    )
+                    if self._db_version >= _PHOTOS_5_VERSION:
+                        # search only original_filename (#594)
+                        photo_list.extend(
+                            [p for p in photos if n in p.original_filename]
+                        )
+                    else:
+                        photo_list.extend(
+                            [
+                                p
+                                for p in photos
+                                if n in p.filename or n in p.original_filename
+                            ]
+                        )
             photos = photo_list
 
         if options.min_size:

osxphotos/utils.py

@@ -17,7 +17,7 @@ import sys
 import unicodedata
 import urllib.parse
 from plistlib import load as plistload
-from typing import Callable, List, Union
+from typing import Callable, List, Union, Optional
 
 import CoreFoundation
 import objc
@@ -28,7 +28,6 @@ from ._constants import UNICODE_FORMAT
 __all__ = [
     "dd_to_dms_str",
     "expand_and_validate_filepath",
-    "findfiles",
     "get_last_library_path",
     "get_system_library_path",
     "increment_filename_with_count",
@@ -266,7 +265,9 @@ def list_photo_libraries():
     # On older MacOS versions, mdfind appears to ignore some libraries
     # glob to find libraries in ~/Pictures then mdfind to find all the others
     # TODO: make this more robust
-    lib_list = glob.glob(f"{pathlib.Path.home()}/Pictures/*.photoslibrary")
+    lib_list = list_directory(
+        f"{pathlib.Path.home()}/Pictures/", glob="*.photoslibrary"
+    )
 
     # On older OS, may not get all libraries so make sure we get the last one
     last_lib = get_last_library_path()
@@ -290,27 +291,90 @@ def normalize_fs_path(path: str) -> str:
     return unicodedata.normalize("NFD", path)
 
 
-def findfiles(pattern, path):
-    """Returns list of filenames from path matched by pattern
-    shell pattern. Matching is case-insensitive.
-    If 'path_' is invalid/doesn't exist, returns []."""
-    if not os.path.isdir(path):
+# def findfiles(pattern, path):
+#     """Returns list of filenames from path matched by pattern
+#     shell pattern. Matching is case-insensitive.
+#     If 'path_' is invalid/doesn't exist, returns []."""
+#     if not os.path.isdir(path):
+#         return []
+
+#     # paths need to be normalized for unicode as filesystem returns unicode in NFD form
+#     pattern = normalize_fs_path(pattern)
+#     rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
+#     files = os.listdir(path)
+#     return [name for name in files if rule.match(name)]
+
+
+def list_directory(
+    directory: Union[str, pathlib.Path],
+    startswith: Optional[str] = None,
+    endswith: Optional[str] = None,
+    contains: Optional[str] = None,
+    glob: Optional[str] = None,
+    include_path: bool = False,
+    case_sensitive: bool = False,
+) -> List[Union[str, pathlib.Path]]:
+    """List directory contents and return list of files or directories matching search criteria.
+    Accounts for case-insensitive filesystems, unicode filenames. directory can be a str or a pathlib.Path object.
+
+    Args:
+        directory: directory to search
+        startswith: string to match at start of filename
+        endswith: string to match at end of filename
+        contains: string to match anywhere in filename
+        glob: shell-style glob pattern to match filename
+        include_path: if True, return full path to file
+        case_sensitive: if True, match case-sensitively
+
+    Returns: List of files or directories matching search criteria as either str or pathlib.Path objects depending on the input type;
+    returns empty list if directory is invalid or doesn't exist.
+
+    """
+    is_pathlib = isinstance(directory, pathlib.Path)
+    if is_pathlib:
+        directory = str(directory)
+
+    if not os.path.isdir(directory):
         return []
 
-    # paths need to be normalized for unicode as filesystem returns unicode in NFD form
-    pattern = normalize_fs_path(pattern)
-    rule = re.compile(fnmatch.translate(pattern), re.IGNORECASE)
-    files = os.listdir(path)
-    return [name for name in files if rule.match(name)]
+    startswith = normalize_fs_path(startswith) if startswith else None
+    endswith = normalize_fs_path(endswith) if endswith else None
+    contains = normalize_fs_path(contains) if contains else None
+    glob = normalize_fs_path(glob) if glob else None
 
+    files = [normalize_fs_path(f) for f in os.listdir(directory)]
+    if not case_sensitive:
+        files_normalized = {f.lower(): f for f in files}
+        files = [f.lower() for f in files]
+        startswith = startswith.lower() if startswith else None
+        endswith = endswith.lower() if endswith else None
+        contains = contains.lower() if contains else None
+        glob = glob.lower() if glob else None
+    else:
+        files_normalized = {f: f for f in files}
 
-def list_directory_startswith(directory_path: str, startswith: str) -> List[str]:
-    """List directory contents and return list of files starting with startswith; returns [] if directory doesn't exist"""
-    if not os.path.isdir(directory_path):
-        return []
-    startswith = normalize_fs_path(startswith)
-    files = [normalize_fs_path(f) for f in os.listdir(directory_path)]
-    return [f for f in files if f.startswith(startswith)]
+    if startswith:
+        files = [f for f in files if f.startswith(startswith)]
+    if endswith:
+        endswith = normalize_fs_path(endswith)
+        files = [f for f in files if f.endswith(endswith)]
+    if contains:
+        contains = normalize_fs_path(contains)
+        files = [f for f in files if contains in f]
+    if glob:
+        glob = normalize_fs_path(glob)
+        flags = re.IGNORECASE if not case_sensitive else 0
+        rule = re.compile(fnmatch.translate(glob), flags)
+        files = [f for f in files if rule.match(f)]
+
+    files = [files_normalized[f] for f in files]
+
+    if include_path:
+        files = [os.path.join(directory, f) for f in files]
+    if is_pathlib:
+        files = [pathlib.Path(f) for f in files]
+
+    return files
 
 
 def _open_sql_file(dbname):
@@ -381,8 +445,8 @@ def increment_filename_with_count(
     Note: This obviously is subject to race condition so using with caution.
     """
     dest = filepath if isinstance(filepath, pathlib.Path) else pathlib.Path(filepath)
-    dest_files = list_directory_startswith(str(dest.parent), dest.stem)
-    dest_files = [pathlib.Path(f).stem.lower() for f in dest_files]
+    dest_files = list_directory(dest.parent, startswith=dest.stem)
+    dest_files = [f.stem.lower() for f in dest_files]
     dest_new = f"{dest.stem} ({count})" if count else dest.stem
     dest_new = normalize_fs_path(dest_new)
 

tests/Test-10.15.7.photoslibrary/database/Photos.sqlite.lock

@@ -7,7 +7,7 @@
 	<key>hostuuid</key>
 	<string>585B80BF-8D1F-55EF-A9E8-6CF4E5523959</string>
 	<key>pid</key>
-	<integer>1961</integer>
+	<integer>14817</integer>
 	<key>processname</key>
 	<string>photolibraryd</string>
 	<key>uid</key>

tests/Test-10.15.7.photoslibrary/private/com.apple.photoanalysisd/caches/graph/PhotoAnalysisServicePreferences.plist

@@ -3,24 +3,24 @@
 <plist version="1.0">
 <dict>
 	<key>BackgroundHighlightCollection</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundHighlightEnrichment</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundJobAssetRevGeocode</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundJobSearch</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>BackgroundPeopleSuggestion</key>
-	<date>2021-09-14T04:40:41Z</date>
+	<date>2022-02-04T13:51:39Z</date>
 	<key>BackgroundUserBehaviorProcessor</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphConsistencyUpdateJobDateKey</key>
 	<date>2021-07-20T05:48:08Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundGraphRebuildJobDate</key>
 	<date>2021-07-20T05:47:59Z</date>
 	<key>PhotoAnalysisGraphLastBackgroundMemoryGenerationJobDate</key>
-	<date>2021-09-14T04:40:43Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 	<key>SiriPortraitDonation</key>
-	<date>2021-09-14T04:40:42Z</date>
+	<date>2022-02-04T13:51:40Z</date>
 </dict>
 </plist>

tests/test_albums_folders_catalina_10_15_7.py

@@ -21,6 +21,7 @@ FOLDER_ALBUM_DICT = {
 ALBUM_NAMES = [
     "2018-10 - Sponsion, Museum, Frühstück, Römermuseum",
     "2019-10/11 Paris Clermont",
+    "Água",
     "AlbumInFolder",
     "EmptyAlbum",
     "I have a deleted twin",
@@ -38,6 +39,7 @@ ALBUM_NAMES = [
 ALBUM_PARENT_DICT = {
     "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": None,
     "2019-10/11 Paris Clermont": None,
+    "Água": None,
     "AlbumInFolder": "SubFolder2",
     "EmptyAlbum": None,
     "I have a deleted twin": None,
@@ -54,6 +56,7 @@ ALBUM_PARENT_DICT = {
 ALBUM_FOLDER_NAMES_DICT = {
     "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": [],
     "2019-10/11 Paris Clermont": [],
+    "Água": [],
     "AlbumInFolder": ["Folder1", "SubFolder2"],
     "EmptyAlbum": [],
     "I have a deleted twin": [],
@@ -70,6 +73,7 @@ ALBUM_FOLDER_NAMES_DICT = {
 ALBUM_LEN_DICT = {
     "2018-10 - Sponsion, Museum, Frühstück, Römermuseum": 1,
     "2019-10/11 Paris Clermont": 1,
+    "Água": 3,
     "AlbumInFolder": 2,
     "EmptyAlbum": 0,
     "I have a deleted twin": 1,
@@ -103,6 +107,11 @@ ALBUM_PHOTO_UUID_DICT = {
         "4D521201-92AC-43E5-8F7C-59BC41C37A96",
         "8E1D7BC9-9321-44F9-8CFB-4083F6B9232A",
     ],
+    "Água": [
+        "7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091",
+        "2DFD33F1-A5D8-486F-A3A9-98C07995535A",
+        "54E76FCB-D353-4557-9997-0A457BCB4D48",
+    ],
 }
 
 UUID_DICT = {

tests/test_catalina_10_15_7.py

@@ -24,10 +24,10 @@ PHOTOS_DB = "tests/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_DB_PATH = "/Test-10.15.7.photoslibrary/database/photos.db"
 PHOTOS_LIBRARY_PATH = "/Test-10.15.7.photoslibrary"
 
-PHOTOS_DB_LEN = 25
-PHOTOS_NOT_IN_TRASH_LEN = 23
+PHOTOS_DB_LEN = 29
+PHOTOS_NOT_IN_TRASH_LEN = 27
 PHOTOS_IN_TRASH_LEN = 2
-PHOTOS_DB_IMPORT_SESSIONS = 17
+PHOTOS_DB_IMPORT_SESSIONS = 21
 
 KEYWORDS = [
     "Kids",
@@ -72,6 +72,7 @@ ALBUMS = [
     "Sorted Oldest First",
     "Sorted Title",
     "Test Album",  # there are 2 albums named "Test Album" for testing duplicate album names
+    "Água",
 ]
 KEYWORDS_DICT = {
     "Drink": 2,
@@ -115,6 +116,7 @@ ALBUM_DICT = {
     "Sorted Oldest First": 3,
     "Sorted Title": 3,
     "Test Album": 2,
+    "Água": 3,
 }  # Note: there are 2 albums named "Test Album" for testing duplicate album names
 
 UUID_DICT = {
@@ -1091,7 +1093,7 @@ def test_from_to_date(photosdb):
     time.tzset()
 
     photos = photosdb.photos(from_date=datetime.datetime(2018, 10, 28))
-    assert len(photos) == 16
+    assert len(photos) == 20
 
     photos = photosdb.photos(to_date=datetime.datetime(2018, 10, 28))
     assert len(photos) == 7

tests/test_cli.py

@@ -8,6 +8,7 @@ from click.testing import CliRunner
 
 import osxphotos
 from osxphotos.exiftool import get_exiftool_path
+from osxphotos.utils import normalize_unicode
 
 CLI_PHOTOS_DB = "tests/Test-10.15.7.photoslibrary"
 LIVE_PHOTOS_DB = "tests/Test-Cloud-10.15.1.photoslibrary"
@@ -79,64 +80,69 @@ CLI_OUTPUT_NO_SUBCOMMAND = [
 CLI_OUTPUT_QUERY_UUID = '[{"uuid": "D79B8D77-BFFC-460B-9312-034F2877D35B", "filename": "D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg", "original_filename": "Pumkins2.jpg", "date": "2018-09-28T16:07:07-04:00", "description": "Girl holding pumpkin", "title": "I found one!", "keywords": ["Kids"], "albums": ["Pumpkin Farm", "Test Album", "Multi Keyword"], "persons": ["Katie"], "path": "/tests/Test-10.15.7.photoslibrary/originals/D/D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg", "ismissing": false, "hasadjustments": false, "external_edit": false, "favorite": false, "hidden": false, "latitude": 41.256566, "longitude": -95.940257, "path_edited": null, "shared": false, "isphoto": true, "ismovie": false, "uti": "public.jpeg", "burst": false, "live_photo": false, "path_live_photo": null, "iscloudasset": false, "incloud": null}]'
 
 CLI_EXPORT_FILENAMES = [
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips.jpg",
-    "wedding.jpg",
-    "wedding_edited.jpeg",
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.dng",
-    "IMG_1693.tif",
-    "IMG_1994.JPG",
-    "IMG_1994.cr2",
-    "IMG_1997.JPG",
-    "IMG_1997.cr2",
-    "IMG_3092.heic",
-    "IMG_3092_edited.jpeg",
-    "IMG_4547.jpg",
-    "Jellyfish.MOV",
-    "Jellyfish1.mp4",
-    "Tulips_edited.jpeg",
-    "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
-    "winebottle (1).jpeg",
-    "Frítest.jpg",
     "Frítest (1).jpg",
     "Frítest (2).jpg",
     "Frítest (3).jpg",
-    "Frítest_edited.jpeg",
     "Frítest_edited (1).jpeg",
+    "Frítest_edited.jpeg",
+    "Frítest.jpg",
+    "IMG_1693.tif",
+    "IMG_1994.cr2",
+    "IMG_1994.JPG",
+    "IMG_1997.cr2",
+    "IMG_1997.JPG",
+    "IMG_3092_edited.jpeg",
+    "IMG_3092.heic",
+    "IMG_4547.jpg",
+    "Jellyfish.MOV",
+    "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
+    "screenshot-really-a-png.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips.jpg",
+    "wedding_edited.jpeg",
+    "wedding.jpg",
+    "winebottle (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 
 CLI_EXPORT_FILENAMES_DRY_RUN = [
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips.jpg",
-    "wedding.jpg",
-    "wedding_edited.jpeg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.dng",
+    "Frítest_edited.jpeg",
+    "Frítest.jpg",
     "IMG_1693.tif",
-    "IMG_1994.JPG",
     "IMG_1994.cr2",
-    "IMG_1997.JPG",
+    "IMG_1994.JPG",
     "IMG_1997.cr2",
-    "IMG_3092.heic",
+    "IMG_1997.JPG",
     "IMG_3092_edited.jpeg",
+    "IMG_3092.heic",
     "IMG_4547.jpg",
     "Jellyfish.MOV",
     "Jellyfish1.mp4",
-    "Tulips_edited.jpeg",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
     "screenshot-really-a-png.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips.jpg",
+    "wedding_edited.jpeg",
+    "wedding.jpg",
     "winebottle.jpeg",
     "winebottle.jpeg",
-    "Frítest.jpg",
-    "Frítest_edited.jpeg",
 ]
 
 CLI_EXPORT_IGNORE_SIGNATURE_FILENAMES = ["Tulips.jpg", "wedding.jpg"]
@@ -154,225 +160,253 @@ CLI_EXPORT_ORIGINAL_SUFFIX_TEMPLATE = "{edited?_original,}"
 CLI_EXPORT_PREVIEW_SUFFIX = "_lowres"
 
 CLI_EXPORT_FILENAMES_EDITED_SUFFIX = [
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_bearbeiten.jpeg",
-    "Tulips.jpg",
-    "wedding.jpg",
-    "wedding_bearbeiten.jpeg",
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.dng",
-    "IMG_1693.tif",
-    "IMG_1994.JPG",
-    "IMG_1994.cr2",
-    "IMG_1997.JPG",
-    "IMG_1997.cr2",
-    "IMG_3092.heic",
-    "IMG_3092_bearbeiten.jpeg",
-    "IMG_4547.jpg",
-    "Jellyfish.MOV",
-    "Jellyfish1.mp4",
-    "Tulips_bearbeiten.jpeg",
-    "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
-    "winebottle (1).jpeg",
-    "Frítest.jpg",
     "Frítest (1).jpg",
     "Frítest (2).jpg",
     "Frítest (3).jpg",
-    "Frítest_bearbeiten.jpeg",
     "Frítest_bearbeiten (1).jpeg",
+    "Frítest_bearbeiten.jpeg",
+    "Frítest.jpg",
+    "IMG_1693.tif",
+    "IMG_1994.cr2",
+    "IMG_1994.JPG",
+    "IMG_1997.cr2",
+    "IMG_1997.JPG",
+    "IMG_3092_bearbeiten.jpeg",
+    "IMG_3092.heic",
+    "IMG_4547.jpg",
+    "Jellyfish.MOV",
+    "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
+    "screenshot-really-a-png.jpeg",
+    "St James Park_bearbeiten.jpeg",
+    "St James Park.jpg",
+    "Tulips_bearbeiten.jpeg",
+    "Tulips.jpg",
+    "wedding_bearbeiten.jpeg",
+    "wedding.jpg",
+    "winebottle (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_EDITED_SUFFIX_TEMPLATE = [
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips.jpg",
-    "wedding.jpg",
-    "wedding_edited.jpeg",
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.dng",
-    "IMG_1693.tif",
-    "IMG_1994.JPG",
-    "IMG_1994.cr2",
-    "IMG_1997.JPG",
-    "IMG_1997.cr2",
-    "IMG_3092.heic",
-    "IMG_3092_edited.jpeg",
-    "IMG_4547.jpg",
-    "Jellyfish.MOV",
-    "Jellyfish1.mp4",
-    "Tulips_edited.jpeg",
-    "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
-    "winebottle (1).jpeg",
-    "Frítest.jpg",
     "Frítest (1).jpg",
     "Frítest (2).jpg",
     "Frítest (3).jpg",
-    "Frítest_edited.jpeg",
     "Frítest_edited (1).jpeg",
+    "Frítest_edited.jpeg",
+    "Frítest.jpg",
+    "IMG_1693.tif",
+    "IMG_1994.cr2",
+    "IMG_1994.JPG",
+    "IMG_1997.cr2",
+    "IMG_1997.JPG",
+    "IMG_3092_edited.jpeg",
+    "IMG_3092.heic",
+    "IMG_4547.jpg",
+    "Jellyfish.MOV",
+    "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
+    "screenshot-really-a-png.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips.jpg",
+    "wedding_edited.jpeg",
+    "wedding.jpg",
+    "winebottle (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX = [
-    "Pumkins1_original.jpg",
-    "Pumkins2_original.jpg",
-    "Pumpkins3_original.jpg",
-    "St James Park_original.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips_original.jpg",
-    "wedding_original.jpg",
-    "wedding_edited.jpeg",
+    "[2020-08-29] AAF035_original (1).jpg",
+    "[2020-08-29] AAF035_original (2).jpg",
+    "[2020-08-29] AAF035_original (3).jpg",
+    "[2020-08-29] AAF035_original.jpg",
     "DSC03584_original.dng",
-    "IMG_1693_original.tif",
-    "IMG_1994_original.JPG",
-    "IMG_1994_original.cr2",
-    "IMG_1997_original.JPG",
-    "IMG_1997_original.cr2",
-    "IMG_3092_original.heic",
-    "IMG_3092_edited.jpeg",
-    "IMG_4547_original.jpg",
-    "Jellyfish_original.MOV",
-    "Jellyfish1_original.mp4",
-    "Tulips_edited.jpeg",
-    "screenshot-really-a-png_original.jpeg",
-    "winebottle_original.jpeg",
-    "winebottle_original (1).jpeg",
-    "Frítest_original.jpg",
+    "Frítest_edited (1).jpeg",
+    "Frítest_edited.jpeg",
     "Frítest_original (1).jpg",
     "Frítest_original (2).jpg",
     "Frítest_original (3).jpg",
-    "Frítest_edited.jpeg",
-    "Frítest_edited (1).jpeg",
+    "Frítest_original.jpg",
+    "IMG_1693_original.tif",
+    "IMG_1994_original.cr2",
+    "IMG_1994_original.JPG",
+    "IMG_1997_original.cr2",
+    "IMG_1997_original.JPG",
+    "IMG_3092_edited.jpeg",
+    "IMG_3092_original.heic",
+    "IMG_4547_original.jpg",
+    "Jellyfish_original.MOV",
+    "Jellyfish1_original.mp4",
+    "Pumkins1_original.jpg",
+    "Pumkins2_original.jpg",
+    "Pumpkins3_original.jpg",
+    "screenshot-really-a-png_original.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park_original.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips_original.jpg",
+    "wedding_edited.jpeg",
+    "wedding_original.jpg",
+    "winebottle_original (1).jpeg",
+    "winebottle_original.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_ORIGINAL_SUFFIX_TEMPLATE = [
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park_original.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips_original.jpg",
-    "wedding_original.jpg",
-    "wedding_edited.jpeg",
-    "Tulips_edited.jpeg",
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.dng",
+    "Frítest (1).jpg",
+    "Frítest_edited (1).jpeg",
+    "Frítest_edited.jpeg",
+    "Frítest_original (1).jpg",
+    "Frítest_original.jpg",
+    "Frítest.jpg",
     "IMG_1693.tif",
-    "IMG_1994.JPG",
     "IMG_1994.cr2",
-    "IMG_1997.JPG",
+    "IMG_1994.JPG",
     "IMG_1997.cr2",
-    "IMG_3092_original.heic",
+    "IMG_1997.JPG",
     "IMG_3092_edited.jpeg",
+    "IMG_3092_original.heic",
     "IMG_4547.jpg",
     "Jellyfish.MOV",
     "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
     "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park_original.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips_original.jpg",
+    "wedding_edited.jpeg",
+    "wedding_original.jpg",
     "winebottle (1).jpeg",
-    "Frítest.jpg",
-    "Frítest (1).jpg",
-    "Frítest_original.jpg",
-    "Frítest_edited.jpeg",
-    "Frítest_original (1).jpg",
-    "Frítest_edited (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_CURRENT = [
+    "1793FAAB-DE75-4E25-886C-2BD66C780D6A_edited.jpeg",  # Frítest.jpg
+    "1793FAAB-DE75-4E25-886C-2BD66C780D6A.jpeg",  # Frítest.jpg
     "1EB2B765-0765-43BA-A90C-0D0580E6172C.jpeg",
+    "2DFD33F1-A5D8-486F-A3A9-98C07995535A.jpeg",
+    "35329C57-B963-48D6-BB75-6AFF9370CBBC.mov",
     "3DD2C897-F19E-4CA6-8C22-B027D5A71907.jpeg",
     "4D521201-92AC-43E5-8F7C-59BC41C37A96.cr2",
     "4D521201-92AC-43E5-8F7C-59BC41C37A96.jpeg",
+    "52083079-73D5-4921-AC1B-FE76F279133F.jpeg",
+    "54E76FCB-D353-4557-9997-0A457BCB4D48.jpeg",
+    "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4_edited.jpeg",
     "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4.jpeg",
+    "7783E8E6-9CAC-40F3-BE22-81FB7051C266_edited.jpeg",
+    "7783E8E6-9CAC-40F3-BE22-81FB7051C266.heic",
+    "7F74DD34-5920-4DA3-B284-479887A34F66.jpeg",
+    "7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091.jpeg",
+    "8846E3E6-8AC8-4857-8448-E3D025784410.tiff",
+    "A8266C97-9BAF-4AF4-99F3-0013832869B8.jpeg",  # Frítest.jpg
     "A92D9C26-3A50-4197-9388-CB5F7DB9FA91.cr2",
     "A92D9C26-3A50-4197-9388-CB5F7DB9FA91.jpeg",
-    "D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068.dng",
-    "D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg",
-    "DC99FBDD-7A52-4100-A5BB-344131646C30.jpeg",
-    "DC99FBDD-7A52-4100-A5BB-344131646C30_edited.jpeg",
-    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51.jpeg",
-    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51_edited.jpeg",
-    "F12384F6-CD17-4151-ACBA-AE0E3688539E.jpeg",
-    "35329C57-B963-48D6-BB75-6AFF9370CBBC.mov",
-    "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4_edited.jpeg",
-    "7783E8E6-9CAC-40F3-BE22-81FB7051C266.heic",
-    "7783E8E6-9CAC-40F3-BE22-81FB7051C266_edited.jpeg",
-    "7F74DD34-5920-4DA3-B284-479887A34F66.jpeg",
-    "8846E3E6-8AC8-4857-8448-E3D025784410.tiff",
-    "D1359D09-1373-4F3B-B0E3-1A4DE573E4A3.mp4",
-    "E2078879-A29C-4D6F-BACB-E3BBE6C3EB91.jpeg",
-    "52083079-73D5-4921-AC1B-FE76F279133F.jpeg",
     "B13F4485-94E0-41CD-AF71-913095D62E31.jpeg",  # Frítest.jpg
-    "1793FAAB-DE75-4E25-886C-2BD66C780D6A.jpeg",  # Frítest.jpg
-    "1793FAAB-DE75-4E25-886C-2BD66C780D6A_edited.jpeg",  # Frítest.jpg
-    "A8266C97-9BAF-4AF4-99F3-0013832869B8.jpeg",  # Frítest.jpg
-    "D1D4040D-D141-44E8-93EA-E403D9F63E07.jpeg",  # Frítest.jpg
+    "D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068.dng",
+    "D1359D09-1373-4F3B-B0E3-1A4DE573E4A3.mp4",
     "D1D4040D-D141-44E8-93EA-E403D9F63E07_edited.jpeg",  # Frítest.jpg
+    "D1D4040D-D141-44E8-93EA-E403D9F63E07.jpeg",  # Frítest.jpg
+    "D79B8D77-BFFC-460B-9312-034F2877D35B.jpeg",
+    "DC99FBDD-7A52-4100-A5BB-344131646C30_edited.jpeg",
+    "DC99FBDD-7A52-4100-A5BB-344131646C30.jpeg",
+    "E2078879-A29C-4D6F-BACB-E3BBE6C3EB91.jpeg",
+    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51_edited.jpeg",
+    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51.jpeg",
+    "F12384F6-CD17-4151-ACBA-AE0E3688539E.jpeg",
+    "F207D5DE-EFAD-4217-8424-0764AAC971D0.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_CONVERT_TO_JPEG = [
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.jpeg",
-    "IMG_1693.jpeg",
-    "IMG_1994.JPG",
-    "IMG_1994.cr2",
-    "IMG_1997.JPG",
-    "IMG_1997.cr2",
-    "IMG_3092.jpeg",
-    "IMG_3092_edited.jpeg",
-    "IMG_4547.jpg",
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips.jpg",
-    "Tulips_edited.jpeg",
-    "wedding.jpg",
-    "wedding_edited.jpeg",
-    "Jellyfish.MOV",
-    "Jellyfish1.mp4",
-    "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
-    "winebottle (1).jpeg",
-    "Frítest.jpg",
     "Frítest (1).jpg",
     "Frítest (2).jpg",
     "Frítest (3).jpg",
     "Frítest_edited (1).jpeg",
     "Frítest_edited.jpeg",
+    "Frítest.jpg",
+    "IMG_1693.jpeg",
+    "IMG_1994.cr2",
+    "IMG_1994.JPG",
+    "IMG_1997.cr2",
+    "IMG_1997.JPG",
+    "IMG_3092_edited.jpeg",
+    "IMG_3092.jpeg",
+    "IMG_4547.jpg",
+    "Jellyfish.MOV",
+    "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
+    "screenshot-really-a-png.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips.jpg",
+    "wedding_edited.jpeg",
+    "wedding.jpg",
+    "winebottle (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 CLI_EXPORT_FILENAMES_CONVERT_TO_JPEG_SKIP_RAW = [
+    "[2020-08-29] AAF035 (1).jpg",
+    "[2020-08-29] AAF035 (2).jpg",
+    "[2020-08-29] AAF035 (3).jpg",
+    "[2020-08-29] AAF035.jpg",
     "DSC03584.jpeg",
-    "IMG_1693.jpeg",
-    "IMG_1994.JPG",
-    "IMG_1997.JPG",
-    "IMG_3092.jpeg",
-    "IMG_3092_edited.jpeg",
-    "IMG_4547.jpg",
-    "Pumkins1.jpg",
-    "Pumkins2.jpg",
-    "Pumpkins3.jpg",
-    "St James Park.jpg",
-    "St James Park_edited.jpeg",
-    "Tulips.jpg",
-    "Tulips_edited.jpeg",
-    "wedding.jpg",
-    "wedding_edited.jpeg",
-    "Jellyfish.MOV",
-    "Jellyfish1.mp4",
-    "screenshot-really-a-png.jpeg",
-    "winebottle.jpeg",
-    "winebottle (1).jpeg",
-    "Frítest.jpg",
     "Frítest (1).jpg",
     "Frítest (2).jpg",
     "Frítest (3).jpg",
-    "Frítest_edited.jpeg",
     "Frítest_edited (1).jpeg",
+    "Frítest_edited.jpeg",
+    "Frítest.jpg",
+    "IMG_1693.jpeg",
+    "IMG_1994.JPG",
+    "IMG_1997.JPG",
+    "IMG_3092_edited.jpeg",
+    "IMG_3092.jpeg",
+    "IMG_4547.jpg",
+    "Jellyfish.MOV",
+    "Jellyfish1.mp4",
+    "Pumkins1.jpg",
+    "Pumkins2.jpg",
+    "Pumpkins3.jpg",
+    "screenshot-really-a-png.jpeg",
+    "St James Park_edited.jpeg",
+    "St James Park.jpg",
+    "Tulips_edited.jpeg",
+    "Tulips.jpg",
+    "wedding_edited.jpeg",
+    "wedding.jpg",
+    "winebottle (1).jpeg",
+    "winebottle.jpeg",
 ]
 
 CLI_EXPORT_CONVERT_TO_JPEG_LARGE_FILE = "DSC03584.jpeg"
@@ -546,7 +580,7 @@ PHOTOS_NOT_IN_TRASH_LEN_14_6 = 12
 PHOTOS_IN_TRASH_LEN_14_6 = 1
 PHOTOS_MISSING_14_6 = 1
 
-PHOTOS_NOT_IN_TRASH_LEN_15_7 = 23
+PHOTOS_NOT_IN_TRASH_LEN_15_7 = 27
 PHOTOS_IN_TRASH_LEN_15_7 = 2
 PHOTOS_MISSING_15_7 = 2
 PHOTOS_EDITED_15_7 = 6
@@ -732,6 +766,7 @@ ALBUMS_JSON = {
         "Sorted Newest First": 3,
         "Sorted Oldest First": 3,
         "Sorted Title": 3,
+        "Água": 3,
     },
     "shared albums": {},
 }
@@ -746,6 +781,7 @@ ALBUMS_STR = """albums:
   2018-10 - Sponsion, Museum, Frühstück, Römermuseum: 1
   2019-10/11 Paris Clermont: 1
   EmptyAlbum: 0
+  Água: 3
 shared albums: {}
 """
 
@@ -820,37 +856,45 @@ UUID_IS_REFERENCE = [
 ]
 
 UUID_IN_ALBUM = [
-    "F12384F6-CD17-4151-ACBA-AE0E3688539E",
-    "8E1D7BC9-9321-44F9-8CFB-4083F6B9232A",
     "1EB2B765-0765-43BA-A90C-0D0580E6172C",
-    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51",
-    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91",
-    "D79B8D77-BFFC-460B-9312-034F2877D35B",
-    "4D521201-92AC-43E5-8F7C-59BC41C37A96",
-    "D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068",
+    "2DFD33F1-A5D8-486F-A3A9-98C07995535A",
     "3DD2C897-F19E-4CA6-8C22-B027D5A71907",
+    "4D521201-92AC-43E5-8F7C-59BC41C37A96",
+    "54E76FCB-D353-4557-9997-0A457BCB4D48",
     "7783E8E6-9CAC-40F3-BE22-81FB7051C266",
+    "7FD37B5F-6FAA-4DB1-8A29-BF9C37E38091",
+    "8E1D7BC9-9321-44F9-8CFB-4083F6B9232A",
+    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91",
+    "D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068",
+    "D79B8D77-BFFC-460B-9312-034F2877D35B",
+    "E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51",
+    "F12384F6-CD17-4151-ACBA-AE0E3688539E",
 ]
 
 UUID_NOT_IN_ALBUM = [
-    "A1DD1F98-2ECD-431F-9AC9-5AFEFE2D3A5C",
-    "DC99FBDD-7A52-4100-A5BB-344131646C30",
-    "D1359D09-1373-4F3B-B0E3-1A4DE573E4A3",
-    "E2078879-A29C-4D6F-BACB-E3BBE6C3EB91",
-    "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4",
-    "35329C57-B963-48D6-BB75-6AFF9370CBBC",
-    "8846E3E6-8AC8-4857-8448-E3D025784410",
-    "7F74DD34-5920-4DA3-B284-479887A34F66",
-    "52083079-73D5-4921-AC1B-FE76F279133F",
-    "B13F4485-94E0-41CD-AF71-913095D62E31",  # Frítest.jpg
     "1793FAAB-DE75-4E25-886C-2BD66C780D6A",  # Frítest.jpg
+    "35329C57-B963-48D6-BB75-6AFF9370CBBC",
+    "52083079-73D5-4921-AC1B-FE76F279133F",
+    "6191423D-8DB8-4D4C-92BE-9BBBA308AAC4",
+    "7F74DD34-5920-4DA3-B284-479887A34F66",
+    "8846E3E6-8AC8-4857-8448-E3D025784410",
+    "A1DD1F98-2ECD-431F-9AC9-5AFEFE2D3A5C",
     "A8266C97-9BAF-4AF4-99F3-0013832869B8",  # Frítest.jpg
+    "B13F4485-94E0-41CD-AF71-913095D62E31",  # Frítest.jpg
+    "D1359D09-1373-4F3B-B0E3-1A4DE573E4A3",
     "D1D4040D-D141-44E8-93EA-E403D9F63E07",  # Frítest.jpg
+    "DC99FBDD-7A52-4100-A5BB-344131646C30",
+    "E2078879-A29C-4D6F-BACB-E3BBE6C3EB91",
+    "F207D5DE-EFAD-4217-8424-0764AAC971D0",
 ]
 
 UUID_DUPLICATES = [
-    "7F74DD34-5920-4DA3-B284-479887A34F66",
+    "2DFD33F1-A5D8-486F-A3A9-98C07995535A",
     "52083079-73D5-4921-AC1B-FE76F279133F",
+    "54E76FCB-D353-4557-9997-0A457BCB4D48",
+    "7F74DD34-5920-4DA3-B284-479887A34F66",
+    "A92D9C26-3A50-4197-9388-CB5F7DB9FA91",
+    "F207D5DE-EFAD-4217-8424-0764AAC971D0",
 ]
 
 UUID_LOCATION = "D79B8D77-BFFC-460B-9312-034F2877D35B"  # Pumkins2.jpg
@@ -2517,7 +2561,8 @@ def test_export_duplicate():
     # pylint: disable=not-context-manager
     with runner.isolated_filesystem():
         result = runner.invoke(
-            export, [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--duplicate"]
+            export,
+            [os.path.join(cwd, CLI_PHOTOS_DB), ".", "-V", "--duplicate", "--skip-raw"],
         )
         assert result.exit_code == 0
         files = glob.glob("*")
@@ -4700,7 +4745,14 @@ def test_export_live_edited():
         # basic export
         result = runner.invoke(
             export,
-            [os.path.join(cwd, PHOTOS_DB_RHET), ".", "-V", "--uuid", UUID_LIVE_EDITED],
+            [
+                os.path.join(cwd, PHOTOS_DB_RHET),
+                ".",
+                "-V",
+                "--uuid",
+                UUID_LIVE_EDITED,
+                "--download-missing",
+            ],
         )
         assert result.exit_code == 0
         files = glob.glob("*")
@@ -5077,7 +5129,7 @@ def test_export_dry_run():
             in result.output
         )
         for filepath in CLI_EXPORT_FILENAMES_DRY_RUN:
-            assert re.search(r"Exported.*" + f"{filepath}", result.output)
+            assert re.search(r"Exported.*" + f"{re.escape(filepath)}", result.output)
             assert not os.path.isfile(normalize_fs_path(filepath))
 
 
@@ -6022,7 +6074,7 @@ def test_export_cleanup_empty_album():
 
 
 def test_export_cleanup_accented_album_name():
-    """test export with --cleanup flag and photos in album with accented unicode characters (#561)"""
+    """test export with --cleanup flag and photos in album with accented unicode characters (#561, #618)"""
     import pathlib
 
     from osxphotos.cli import export
@@ -6045,6 +6097,86 @@ def test_export_cleanup_accented_album_name():
         )
         assert "Deleted: 0 files, 0 directories" in result.output
 
+        # do it again
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                tempdir,
+                "-V",
+                "--update",
+                "--cleanup",
+                "--directory",
+                "{folder_album}",
+                "--update",
+            ],
+        )
+        assert "exported: 0, updated: 0" in result.output
+        assert "Deleted: 0 files, 0 directories" in result.output
+
+
+@pytest.mark.skipif(exiftool is None, reason="exiftool not installed")
+def test_export_cleanup_exiftool_accented_album_name_same_filenames():
+    """test export with --cleanup flag and photos in album with accented unicode characters (#561, #618)"""
+    import pathlib
+
+    from osxphotos.cli import export
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    # pylint: disable=not-context-manager
+    with tempfile.TemporaryDirectory() as tempdir:
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                tempdir,
+                "-V",
+                "--cleanup",
+                "--directory",
+                "{album[/,.|:,.]}",
+                "--exiftool",
+                "--exiftool-merge-keywords",
+                "--exiftool-merge-persons",
+                "--keyword-template",
+                "{keyword}",
+                "--report",
+                "test.csv",
+                "--skip-original-if-edited",
+                "--update",
+                "--touch-file",
+                "--not-hidden",
+            ],
+        )
+        assert "Deleted: 0 files, 0 directories" in result.output
+
+        # do it again
+        result = runner.invoke(
+            export,
+            [
+                os.path.join(cwd, CLI_PHOTOS_DB),
+                tempdir,
+                "-V",
+                "--cleanup",
+                "--directory",
+                "{album[/,.|:,.]}",
+                "--exiftool",
+                "--exiftool-merge-keywords",
+                "--exiftool-merge-persons",
+                "--keyword-template",
+                "{keyword}",
+                "--report",
+                "test.csv",
+                "--skip-original-if-edited",
+                "--update",
+                "--touch-file",
+                "--not-hidden",
+            ],
+        )
+        assert "exported: 0, updated: 0" in result.output
+        assert "updated EXIF data: 0" in result.output
+        assert "Deleted: 0 files, 0 directories" in result.output
+
 
 def test_save_load_config():
     """test --save-config, --load-config"""
@@ -7001,6 +7133,30 @@ def test_query_name():
     assert json_got[0]["original_filename"] == "DSC03584.dng"
 
 
+def test_query_name_unicode():
+    """test query --name with a unicode name"""
+    import json
+    import os
+    import os.path
+
+    import osxphotos
+    from osxphotos.cli import query
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    result = runner.invoke(
+        query,
+        ["--json", "--db", os.path.join(cwd, PHOTOS_DB_15_7), "--name", "Frítest"],
+    )
+    assert result.exit_code == 0
+    json_got = json.loads(result.output)
+
+    assert len(json_got) == 4
+    assert normalize_unicode(json_got[0]["original_filename"]).startswith(
+        normalize_unicode("Frítest.jpg")
+    )
+
+
 def test_query_name_i():
     """test query --name -i"""
     import json
@@ -7030,6 +7186,46 @@ def test_query_name_i():
     assert json_got[0]["original_filename"] == "DSC03584.dng"
 
 
+def test_query_name_original_filename():
+    """test query --name only searches original filename on Photos 5+"""
+    import json
+    import os
+    import os.path
+
+    from osxphotos.cli import query
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    result = runner.invoke(
+        query,
+        ["--json", "--db", os.path.join(cwd, PHOTOS_DB_15_7), "--name", "AA"],
+    )
+    assert result.exit_code == 0
+    json_got = json.loads(result.output)
+
+    assert len(json_got) == 4
+
+
+def test_query_name_original_filename_i():
+    """test query --name only searches original filename on Photos 5+ with -i"""
+    import json
+    import os
+    import os.path
+
+    from osxphotos.cli import query
+
+    runner = CliRunner()
+    cwd = os.getcwd()
+    result = runner.invoke(
+        query,
+        ["--json", "--db", os.path.join(cwd, PHOTOS_DB_15_7), "--name", "aa", "-i"],
+    )
+    assert result.exit_code == 0
+    json_got = json.loads(result.output)
+
+    assert len(json_got) == 4
+
+
 def test_export_name():
     """test export --name"""
     import glob

tests/test_utils.py

@@ -1,27 +1,33 @@
+import logging
+import os.path
+import pathlib
+import tempfile
+
 import pytest
 
+import osxphotos
+
 DB_LOCKED_10_12 = "./tests/Test-Lock-10_12.photoslibrary/database/photos.db"
 DB_LOCKED_10_15 = "./tests/Test-Lock-10_15_1.photoslibrary/database/Photos.sqlite"
 DB_UNLOCKED_10_15 = "./tests/Test-10.15.1.photoslibrary/database/photos.db"
 
 UTI_DICT = {"public.jpeg": "jpeg", "com.canon.cr2-raw-image": "cr2"}
 
+from osxphotos.utils import (
+    _dd_to_dms,
+    increment_filename,
+    increment_filename_with_count,
+    list_directory,
+)
+
 
 def test_debug_enable():
-    import logging
-
-    import osxphotos
-
     osxphotos._set_debug(True)
     logger = osxphotos._get_logger()
     assert logger.isEnabledFor(logging.DEBUG)
 
 
 def test_debug_disable():
-    import logging
-
-    import osxphotos
-
     osxphotos._set_debug(False)
     logger = osxphotos._get_logger()
     assert not logger.isEnabledFor(logging.DEBUG)
@@ -29,14 +35,12 @@ def test_debug_disable():
 
 def test_dd_to_dms():
     # expands coverage for edge case in _dd_to_dms
-    from osxphotos.utils import _dd_to_dms
 
     assert _dd_to_dms(-0.001) == (0, 0, -3.6)
 
 
 @pytest.mark.skip(reason="Fails on some machines")
 def test_get_system_library_path():
-    import osxphotos
 
     _, major, _ = osxphotos.utils._get_os_version()
     if int(major) < 15:
@@ -46,51 +50,73 @@ def test_get_system_library_path():
 
 
 def test_db_is_locked_locked():
-    import osxphotos
 
     assert osxphotos.utils._db_is_locked(DB_LOCKED_10_12)
     assert osxphotos.utils._db_is_locked(DB_LOCKED_10_15)
 
 
 def test_db_is_locked_unlocked():
-    import osxphotos
 
     assert not osxphotos.utils._db_is_locked(DB_UNLOCKED_10_15)
 
 
-def test_findfiles():
-    import os.path
-    import tempfile
-
-    from osxphotos.utils import findfiles
+def test_list_directory():
+    """test list_directory"""
 
     temp_dir = tempfile.TemporaryDirectory(prefix="osxphotos_")
-    fd = open(os.path.join(temp_dir.name, "file1.jpg"), "w+")
-    fd.close
-    fd = open(os.path.join(temp_dir.name, "file2.JPG"), "w+")
-    fd.close
-    files = findfiles("*.jpg", temp_dir.name)
+    temp_dir_name = pathlib.Path(temp_dir.name)
+    file1 = (temp_dir_name / "file1.jpg").touch()
+    file2 = (temp_dir_name / "File2.JPG").touch()
+    file3 = (temp_dir_name / "File.png").touch()
+    file4 = (temp_dir_name / "document.pdf").touch()
+
+    files = list_directory(temp_dir.name, glob="*.jpg")
     assert len(files) == 2
     assert "file1.jpg" in files
-    assert "file2.JPG" in files
+    assert "File2.JPG" in files
+    assert isinstance(files[0], str)
+
+    files = list_directory(temp_dir.name, glob="*.jpg", case_sensitive=True)
+    assert len(files) == 1
+    assert "file1.jpg" in files
+
+    files = list_directory(temp_dir.name, startswith="file")
+    assert len(files) == 3
+
+    files = list_directory(temp_dir.name, endswith="jpg")
+    assert len(files) == 2
+
+    files = list_directory(temp_dir.name, contains="doc")
+    assert len(files) == 1
+    assert "document.pdf" in files
+
+    files = list_directory(temp_dir.name, startswith="File", case_sensitive=True)
+    assert len(files) == 2
+
+    files = list_directory(temp_dir.name, startswith="File", case_sensitive=False)
+    assert len(files) == 3
+
+    files = list_directory(temp_dir.name, startswith="document", include_path=True)
+    assert len(files) == 1
+    assert files[0] == str(pathlib.Path(temp_dir.name) / "document.pdf")
+
+    # test pathlib.Path
+    files = list_directory(temp_dir_name, glob="*.jpg")
+    assert isinstance(files[0], pathlib.Path)
+
+    files = list_directory(temp_dir.name, glob="FooBar*.jpg")
+    assert not files
 
 
-def test_findfiles_invalid_dir():
-    import tempfile
-
-    from osxphotos.utils import findfiles
+def test_list_directory_invalid():
 
     temp_dir = tempfile.TemporaryDirectory(prefix="osxphotos_")
-    files = findfiles("*.jpg", f"{temp_dir.name}/no_such_dir")
+    files = list_directory(f"{temp_dir.name}/no_such_dir", glob="*.jpg")
     assert len(files) == 0
 
 
 def test_increment_filename():
     # test that increment_filename works
-    import pathlib
-    import tempfile
-
-    from osxphotos.utils import increment_filename, increment_filename_with_count
 
     with tempfile.TemporaryDirectory(prefix="osxphotos_") as temp_dir:
         temp_dir = pathlib.Path(temp_dir)