bn8/python-mylib
1
0
Fork 0
Code Issues Pull requests Projects Releases 12 Wiki Activity Actions

Introduce some new pre-commit hooks

Browse source
This commit is contained in:
Benjamin Renard 2024-03-15 09:52:23 +01:00
parent 09c422efe2
commit 85caf81ac2
22 changed files with 183 additions and 115 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

47
.pre-commit-config.yaml
View file

@ -1,31 +1,58 @@
# Pre-commit hooks to run tests and ensure code is cleaned.
# See https://pre-commit.com for more information
---
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.6
hooks:
- id: ruff
args: ["--fix"]
- repo: https://github.com/asottile/pyupgrade
rev: v3.3.1
rev: v3.15.0
hooks:
- id: pyupgrade
args: ['--keep-percent-format', '--py37-plus']
args: ["--keep-percent-format", "--py37-plus"]
- repo: https://github.com/psf/black
rev: 22.12.0
rev: 23.11.0
hooks:
- id: black
args: ['--target-version', 'py37', '--line-length', '100']
args: ["--target-version", "py37", "--line-length", "100"]
- repo: https://github.com/PyCQA/isort
rev: 5.11.5
rev: 5.12.0
hooks:
- id: isort
args: ['--profile', 'black', '--line-length', '100']
args: ["--profile", "black", "--line-length", "100"]
- repo: https://github.com/PyCQA/flake8
rev: 6.0.0
rev: 6.1.0
hooks:
- id: flake8
args: ['--max-line-length=100']
args: ["--max-line-length=100"]
- repo: https://github.com/codespell-project/codespell
rev: v2.2.2
hooks:
- id: codespell
args:
- --ignore-words-list=exten
- --skip="./.*,*.csv,*.json,*.ini,*.subject,*.txt,*.html,*.log,*.conf"
- --quiet-level=2
- --ignore-regex=.*codespell-ignore$
# - --write-changes # Uncomment to write changes
exclude_types: [csv, json]
- repo: https://github.com/adrienverge/yamllint
rev: v1.32.0
hooks:
- id: yamllint
ignore: .github/
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v2.7.1
hooks:
- id: prettier
args: ["--print-width", "100"]
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint --extension-pkg-whitelist=cx_Oracle
entry: ./.pre-commit-pylint --extension-pkg-whitelist=cx_Oracle
language: system
types: [python]
require_serial: true
@ -38,7 +65,7 @@ repos:
hooks:
- id: pytest
name: pytest
entry: python3 -m pytest tests
entry: ./.pre-commit-pytest tests
language: system
types: [python]
pass_filenames: false

21
.pre-commit-pylint Executable file
View file

@ -0,0 +1,21 @@
#!/bin/bash
PWD=`pwd`
if [ -d "$PWD/venv" ]
then
echo "Run pylint inside venv ($PWD/venv)..."
[ ! -e "$PWD/venv/bin/pylint" ] && $PWD/venv/bin/python -m pip install pylint
$PWD/venv/bin/pylint "$@"
exit $?
elif [ -e "$PWD/pyproject.toml" ]
then
echo "Run pylint using poetry..."
poetry run pylint --version > /dev/null 2>&1 || poetry run python -m pip install pylint
poetry run pylint "$@"
exit $?
else
echo "Run pylint at system scope..."
pylint "$@"
exit $?
fi

21
.pre-commit-pytest Executable file
View file

@ -0,0 +1,21 @@
#!/bin/bash
PWD=`pwd`
if [ -d "$PWD/venv" ]
then
echo "Run pytest inside venv ($PWD/venv)..."
[ ! -e "$PWD/venv/bin/pytest" ] && $PWD/venv/bin/python -m pip install pytest
$PWD/venv/bin/pytest "$@"
exit $?
elif [ -e "$PWD/pyproject.toml" ]
then
echo "Run pytest using poetry..."
poetry run pytest --version > /dev/null 2>&1 || poetry run python -m pip install pytest
poetry run pytest "$@"
exit $?
else
echo "Run pytest at system scope..."
pytest "$@"
exit $?
fi

25
README.md
View file

@ -35,35 +35,35 @@ Just run `pip install git+https://gitea.zionetrix.net/bn8/python-mylib.git`
Just run `python setup.py install`
**Note:** This project could previously use as independent python files (not as module). This old version is keep in *legacy* git branch (not maintained).
**Note:** This project could previously use as independent python files (not as module). This old version is keep in _legacy_ git branch (not maintained).
## Include libs
* **mylib.email.EmailClient:** An email client to forge (eventually using template) and send email via a SMTP server
* **mylib.ldap.LdapServer:** A small lib to make requesting LDAP server easier. It's also provide some helper functions to deal with LDAP date string.
* **mylib.mysql.MyDB:** An extra small lib to remember me how to interact with MySQL/MariaDB database
* **mylib.pgsql.PgDB:** An small lib to remember me how to interact with PostgreSQL database. **Warning:** The insert/update/delete/select methods demonstrate how to forge raw SQL request, but **it's a bad idea**: Prefer using prepared query.
* **mylib.opening_hours:** A set of helper functions to deal with french opening hours (including normal opening hours, exceptional closure and nonworking public holidays).
* **mylib.pbar.Pbar:** A small lib for progress bar
* **mylib.report.Report:** A small lib to implement logging based email report send at exit
- **mylib.email.EmailClient:** An email client to forge (eventually using template) and send email via a SMTP server
- **mylib.ldap.LdapServer:** A small lib to make requesting LDAP server easier. It's also provide some helper functions to deal with LDAP date string.
- **mylib.mysql.MyDB:** An extra small lib to remember me how to interact with MySQL/MariaDB database
- **mylib.pgsql.PgDB:** An small lib to remember me how to interact with PostgreSQL database. **Warning:** The insert/update/delete/select methods demonstrate how to forge raw SQL request, but **it's a bad idea**: Prefer using prepared query.
- **mylib.opening_hours:** A set of helper functions to deal with french opening hours (including normal opening hours, exceptional closure and nonworking public holidays).
- **mylib.pbar.Pbar:** A small lib for progress bar
- **mylib.report.Report:** A small lib to implement logging based email report send at exit
To know how to use these libs, you can take a look on *mylib.scripts* content or in *tests* directory.
To know how to use these libs, you can take a look on _mylib.scripts_ content or in _tests_ directory.
## Code Style
[pylint](https://pypi.org/project/pylint/) is used to check for errors and enforces a coding standard, using thoses parameters:
[pylint](https://pypi.org/project/pylint/) is used to check for errors and enforces a coding standard, using those parameters:
```bash
pylint --extension-pkg-whitelist=cx_Oracle
```
[flake8](https://pypi.org/project/flake8/) is also used to check for errors and enforces a coding standard, using thoses parameters:
[flake8](https://pypi.org/project/flake8/) is also used to check for errors and enforces a coding standard, using those parameters:
```bash
flake8 --max-line-length=100
```
[black](https://pypi.org/project/black/) is used to format the code, using thoses parameters:
[black](https://pypi.org/project/black/) is used to format the code, using those parameters:
```bash
black --target-version py37 --line-length 100
@ -83,7 +83,6 @@ pyupgrade --keep-percent-format --py37-plus
**Note:** There is `.pre-commit-config.yaml` to use [pre-commit](https://pre-commit.com/) to automatically run these tools before commits. After cloning the repository, execute `pre-commit install` to install the git hook.
## Copyright
Copyright (c) 2013-2021 Benjamin Renard <brenard@zionetrix.net>

2
build.sh
View file

@ -81,7 +81,7 @@ cd deb_dist/mylib-$VERSION
if [ -z "$DEBIAN_CODENAME" ]
then
echo "Retreive debian codename using lsb_release..."
echo "Retrieve debian codename using lsb_release..."
DEBIAN_CODENAME=$( lsb_release -c -s )
[ $( lsb_release -r -s ) -ge 9 ] && DEBIAN_CODENAME="${DEBIAN_CODENAME}-ee"
else

12
mylib/__init__.py
View file

@ -1,7 +1,7 @@
""" Some really common helper functions """
#
# Pretty formating helpers
# Pretty formatting helpers
#
@ -11,7 +11,7 @@ def increment_prefix(prefix):
def pretty_format_value(value, encoding="utf8", prefix=None):
"""Returned pretty formated value to display"""
"""Returned pretty formatted value to display"""
if isinstance(value, dict):
return pretty_format_dict(value, encoding=encoding, prefix=prefix)
if isinstance(value, list):
@ -27,10 +27,10 @@ def pretty_format_value(value, encoding="utf8", prefix=None):
def pretty_format_value_in_list(value, encoding="utf8", prefix=None):
"""
Returned pretty formated value to display in list
Returned pretty formatted value to display in list
That method will prefix value with line return and incremented prefix
if pretty formated value contains line return.
if pretty formatted value contains line return.
"""
prefix = prefix if prefix else ""
value = pretty_format_value(value, encoding, prefix)
@ -41,7 +41,7 @@ def pretty_format_value_in_list(value, encoding="utf8", prefix=None):
def pretty_format_dict(value, encoding="utf8", prefix=None):
"""Returned pretty formated dict to display"""
"""Returned pretty formatted dict to display"""
prefix = prefix if prefix else ""
result = []
for key in sorted(value.keys()):
@ -53,7 +53,7 @@ def pretty_format_dict(value, encoding="utf8", prefix=None):
def pretty_format_list(row, encoding="utf8", prefix=None):
"""Returned pretty formated list to display"""
"""Returned pretty formatted list to display"""
prefix = prefix if prefix else ""
result = []
for idx, values in enumerate(row):

14
mylib/config.py
View file

@ -465,7 +465,7 @@ class PasswordOption(StringOption):
service_name = self._keyring_service_name
username = self._keyring_username
log.debug("Retreive password %s for username=%s from keyring", service_name, username)
log.debug("Retrieve password %s for username=%s from keyring", service_name, username)
value = keyring.get_password(service_name, username)
if value is None:
@ -757,7 +757,7 @@ class Config: # pylint: disable=too-many-instance-attributes
self.sections[name] = ConfigSection(self, name, **kwargs)
if loaded_callback:
self._loaded_callbacks.append(loaded_callback)
# If configuration is already loaded, execute callback immediatly
# If configuration is already loaded, execute callback immediately
if self._filepath or self.options:
self._loaded()
return self.sections[name]
@ -1155,7 +1155,7 @@ class Config: # pylint: disable=too-many-instance-attributes
dest="validate",
help=(
"Validate configuration: initialize application to test if provided parameters"
" works.\n\nNote: Validation will occured after configuration file creation or"
" works.\n\nNote: Validation will occurred after configuration file creation or"
" update. On error, re-run with -O/--overwrite parameter to fix it."
),
)
@ -1197,7 +1197,7 @@ class Config: # pylint: disable=too-many-instance-attributes
if options.validate:
validate()
else:
print(f"Error occured creating configuration file {options.config}")
print(f"Error occurred creating configuration file {options.config}")
sys.exit(1)
sys.exit(0)
@ -1282,7 +1282,7 @@ class ConfigurableObject:
raise ConfigException(f"No configuration name defined for {__name__}")
def _get_option(self, option, default=None, required=False):
"""Retreive option value"""
"""Retrieve option value"""
if self._kwargs and option in self._kwargs:
return self._kwargs[option]
@ -1302,7 +1302,7 @@ class ConfigurableObject:
def set_default(self, option, default_value):
"""Set option default value"""
assert option in self._defaults, f"Unkown option {option}"
assert option in self._defaults, f"Unknown option {option}"
self._defaults[option] = default_value
def set_defaults(self, **default_values):
@ -1394,7 +1394,7 @@ class ConfigurableObject:
return True
# If Config provided, use it's get_option() method to obtain a global just_try parameter
# value with a defaut to False, otherwise always false
# value with a default to False, otherwise always false
return self._config.get_option("just_try", default=False) if self._config else False

2
mylib/db.py
View file

@ -38,7 +38,7 @@ class DBFailToConnect(DBException, RuntimeError):
"""
def __init__(self, uri):
super().__init__("An error occured during database connection ({uri})", uri=uri)
super().__init__("An error occurred during database connection ({uri})", uri=uri)
class DBDuplicatedSQLParameter(DBException, KeyError):

6
mylib/email.py
View file

@ -239,7 +239,7 @@ class EmailClient(
msg["Date"] = email.utils.formatdate(None, True)
encoding = encoding if encoding else self._get_option("encoding")
if template:
assert template in self.templates, f"Unknwon template {template}"
assert template in self.templates, f"Unknown template {template}"
# Handle subject from template
if not subject:
assert self.templates[template].get(
@ -251,7 +251,7 @@ class EmailClient(
else self.templates[template]["subject"].format(**template_vars)
)
# Put HTML part in last one to prefered it
# Put HTML part in last one to preferred it
parts = []
if self.templates[template].get("text"):
if isinstance(self.templates[template]["text"], MakoTemplate):
@ -322,7 +322,7 @@ class EmailClient(
catch_addr = self._get_option("catch_all_addr")
if catch_addr:
log.debug(
"Catch email originaly send to %s (CC:%s, BCC:%s) to %s",
"Catch email originally send to %s (CC:%s, BCC:%s) to %s",
", ".join(recipients),
", ".join(cc) if isinstance(cc, list) else cc,
", ".join(bcc) if isinstance(bcc, list) else bcc,

26
mylib/ldap.py
View file

@ -211,7 +211,7 @@ class LdapServer:
result_page_control = rctrl
break
# If PagedResultsControl answer not detected, paged serach
# If PagedResultsControl answer not detected, paged search
if not result_page_control:
self._error(
"LdapServer - Server ignores RFC2696 control, paged search can not works",
@ -238,7 +238,7 @@ class LdapServer:
page_control.cookie = result_page_control.cookie
self.logger.debug(
"LdapServer - Paged search end: %d object(s) retreived in %d page(s) of %d object(s)",
"LdapServer - Paged search end: %d object(s) retrieved in %d page(s) of %d object(s)",
len(ret),
pages_count,
pagesize,
@ -379,12 +379,12 @@ class LdapServer:
@staticmethod
def get_dn(obj):
"""Retreive an on object DN from its entry in LDAP search result"""
"""Retrieve an on object DN from its entry in LDAP search result"""
return obj[0][0]
@staticmethod
def get_attr(obj, attr, all_values=None, default=None, decode=False):
"""Retreive an on object attribute value(s) from the object entry in LDAP search result"""
"""Retrieve an on object attribute value(s) from the object entry in LDAP search result"""
if attr not in obj:
for k in obj:
if k.lower() == attr.lower():
@ -437,7 +437,7 @@ class LdapClient:
self.initialize()
def _get_option(self, option, default=None, required=False):
"""Retreive option value"""
"""Retrieve option value"""
if self._options and hasattr(self._options, self._options_prefix + option):
return getattr(self._options, self._options_prefix + option)
@ -500,7 +500,7 @@ class LdapClient:
self.config = loaded_config
uri = self._get_option("uri", required=True)
binddn = self._get_option("binddn")
log.info("Connect to LDAP server %s as %s", uri, binddn if binddn else "annonymous")
log.info("Connect to LDAP server %s as %s", uri, binddn if binddn else "anonymous")
self._conn = LdapServer(
uri,
dn=binddn,
@ -553,7 +553,7 @@ class LdapClient:
:param attr: The attribute name
:param all_values: If True, all values of the attribute will be
returned instead of the first value only
(optinal, default: False)
(optional, default: False)
"""
if attr not in obj:
for k in obj:
@ -582,7 +582,7 @@ class LdapClient:
:param name: The object type name
:param filterstr: The LDAP filter to use to search objects on LDAP directory
:param basedn: The base DN of the search
:param attrs: The list of attribute names to retreive
:param attrs: The list of attribute names to retrieve
:param key_attr: The attribute name or 'dn' to use as key in result
(optional, if leave to None, the result will be a list)
:param warn: If True, a warning message will be logged if no object is found
@ -594,7 +594,7 @@ class LdapClient:
(optional, default: see LdapServer.paged_search)
"""
if name in self._cached_objects:
log.debug("Retreived %s objects from cache", name)
log.debug("Retrieved %s objects from cache", name)
else:
assert self._conn or self.initialize()
log.debug(
@ -643,7 +643,7 @@ class LdapClient:
:param object_name: The object name (only use in log messages)
:param filterstr: The LDAP filter to use to search the object on LDAP directory
:param basedn: The base DN of the search
:param attrs: The list of attribute names to retreive
:param attrs: The list of attribute names to retrieve
:param warn: If True, a warning message will be logged if no object is found
in LDAP directory (otherwise, it will be just a debug message)
(optional, default: True)
@ -855,7 +855,7 @@ class LdapClient:
Update an object
:param ldap_obj: The original LDAP object
:param changes: The changes to make on LDAP object (as formated by get_changes() method)
:param changes: The changes to make on LDAP object (as formatted by get_changes() method)
:param protected_attrs: An optional list of protected attributes
:param rdn_attr: The LDAP object RDN attribute (to detect renaming, default: auto-detected)
:param rdn_attr: Enable relax modification server control (optional, default: false)
@ -915,7 +915,7 @@ class LdapClient:
# Otherwise, update object DN
ldap_obj["dn"] = new_dn
else:
log.debug("%s: No change detected on RDN attibute %s", ldap_obj["dn"], rdn_attr)
log.debug("%s: No change detected on RDN attribute %s", ldap_obj["dn"], rdn_attr)
try:
if self._just_try:
@ -1103,7 +1103,7 @@ def format_date(value, from_timezone=None, to_timezone=None, naive=True):
(optional, default : server local timezone)
:param to_timezone: The timezone used in LDAP (optional, default : UTC)
:param naive: Use naive datetime : do not handle timezone conversion before
formating and return datetime as UTC (because LDAP required a
formatting and return datetime as UTC (because LDAP required a
timezone)
"""
assert isinstance(

2
mylib/mapping.py
View file

@ -29,7 +29,7 @@ Mapping configuration
'join': '[glue]', # If present, sources values will be join using the "glue"
# Alternative mapping
'or': { [map configuration] } # If this mapping case does not retreive any value, try to
'or': { [map configuration] } # If this mapping case does not retrieve any value, try to
# get value(s) with this other mapping configuration
},
'[dst key 2]': {

2
mylib/mysql.py
View file

@ -41,7 +41,7 @@ class MyDB(DB):
)
except Error as err:
log.fatal(
"An error occured during MySQL database connection (%s@%s:%s).",
"An error occurred during MySQL database connection (%s@%s:%s).",
self._user,
self._host,
self._db,

2
mylib/oracle.py
View file

@ -31,7 +31,7 @@ class OracleDB(DB):
self._conn = cx_Oracle.connect(user=self._user, password=self._pwd, dsn=self._dsn)
except cx_Oracle.Error as err:
log.fatal(
"An error occured during Oracle database connection (%s@%s).",
"An error occurred during Oracle database connection (%s@%s).",
self._user,
self._dsn,
exc_info=1,

6
mylib/pgsql.py
View file

@ -45,7 +45,7 @@ class PgDB(DB):
)
except psycopg2.Error as err:
log.fatal(
"An error occured during Postgresql database connection (%s@%s, database=%s).",
"An error occurred during Postgresql database connection (%s@%s, database=%s).",
self._user,
self._host,
self._db,
@ -71,7 +71,7 @@ class PgDB(DB):
return True
except psycopg2.Error:
log.error(
'An error occured setting Postgresql database connection encoding to "%s"',
'An error occurred setting Postgresql database connection encoding to "%s"',
enc,
exc_info=1,
)
@ -126,7 +126,7 @@ class PgDB(DB):
return False
#
# Depreated helpers
# Deprecated helpers
#
@classmethod

2
mylib/report.py
View file

@ -96,7 +96,7 @@ class Report(ConfigurableObject): # pylint: disable=useless-object-inheritance
self.send_at_exit()
def get_handler(self):
"""Retreive logging handler"""
"""Retrieve logging handler"""
return self.handler
def write(self, msg):

2
mylib/scripts/helpers.py
View file

@ -31,7 +31,7 @@ def init_logging(options, name, report=None):
def get_default_opt_value(config, default_config, key):
"""Retreive default option value from config or default config dictionaries"""
"""Retrieve default option value from config or default config dictionaries"""
if config and key in config:
return config[key]
return default_config.get(key)

2
mylib/scripts/sftp_test.py
View file

@ -47,7 +47,7 @@ def main(argv=None): # pylint: disable=too-many-locals,too-many-statements
sftp.connect()
atexit.register(sftp.close)
log.debug("Create tempory file")
log.debug("Create temporary file")
test_content = b"Juste un test."
tmp_dir = tempfile.TemporaryDirectory() # pylint: disable=consider-using-with
tmp_file = os.path.join(

4
mylib/sftp.py
View file

@ -116,13 +116,13 @@ class SFTPClient(ConfigurableObject):
if self.initial_directory:
log.debug("Initial remote directory: '%s'", self.initial_directory)
else:
log.debug("Fail to retreive remote directory, use empty string instead")
log.debug("Fail to retrieve remote directory, use empty string instead")
self.initial_directory = ""
def get_file(self, remote_filepath, local_filepath):
"""Retrieve a file from SFTP server"""
self.connect()
log.debug("Retreive file '%s' to '%s'", remote_filepath, local_filepath)
log.debug("Retrieve file '%s' to '%s'", remote_filepath, local_filepath)
return self.sftp_client.get(remote_filepath, local_filepath) is None
def open_file(self, remote_filepath, mode="r"):

2
mylib/telltale.py
View file

@ -35,7 +35,7 @@ class TelltaleFile:
@property
def last_update(self):
"""Retreive last update datetime of the telltall file"""
"""Retrieve last update datetime of the telltall file"""
try:
return datetime.datetime.fromtimestamp(os.stat(self.filepath).st_mtime)
except FileNotFoundError:

2
tests.sh
View file

@ -32,7 +32,7 @@ do
set -x
;;
*)
usage "Unkown parameter '$OPT'"
usage "Unknown parameter '$OPT'"
esac
let idx=idx+1
done

32
tests/test_config.py
View file

@ -10,7 +10,7 @@ import pytest
from mylib.config import BooleanOption, Config, ConfigSection, StringOption
runned = {}
tested = {}
def test_config_init_default_args():
@ -58,24 +58,24 @@ def test_add_section_with_callback():
config = Config("Test app")
name = "test_section"
global runned
runned["test_add_section_with_callback"] = False
global tested
tested["test_add_section_with_callback"] = False
def test_callback(loaded_config):
global runned
global tested
assert loaded_config == config
assert runned["test_add_section_with_callback"] is False
runned["test_add_section_with_callback"] = True
assert tested["test_add_section_with_callback"] is False
tested["test_add_section_with_callback"] = True
section = config.add_section(name, loaded_callback=test_callback)
assert isinstance(section, ConfigSection)
assert test_callback in config._loaded_callbacks
assert runned["test_add_section_with_callback"] is False
assert tested["test_add_section_with_callback"] is False
config.parse_arguments_options(argv=[], create=False)
assert runned["test_add_section_with_callback"] is True
assert tested["test_add_section_with_callback"] is True
assert test_callback in config._loaded_callbacks_executed
# Try to execute again to verify callback is not runned again
# Try to execute again to verify callback is not tested again
config._loaded()
@ -84,21 +84,21 @@ def test_add_section_with_callback_already_loaded():
name = "test_section"
config.parse_arguments_options(argv=[], create=False)
global runned
runned["test_add_section_with_callback_already_loaded"] = False
global tested
tested["test_add_section_with_callback_already_loaded"] = False
def test_callback(loaded_config):
global runned
global tested
assert loaded_config == config
assert runned["test_add_section_with_callback_already_loaded"] is False
runned["test_add_section_with_callback_already_loaded"] = True
assert tested["test_add_section_with_callback_already_loaded"] is False
tested["test_add_section_with_callback_already_loaded"] = True
section = config.add_section(name, loaded_callback=test_callback)
assert isinstance(section, ConfigSection)
assert runned["test_add_section_with_callback_already_loaded"] is True
assert tested["test_add_section_with_callback_already_loaded"] is True
assert test_callback in config._loaded_callbacks
assert test_callback in config._loaded_callbacks_executed
# Try to execute again to verify callback is not runned again
# Try to execute again to verify callback is not tested again
config._loaded()