ioos_qc.config_creator package

class ioos_qc.config_creator.CreatorConfig(path_or_dict, schema={'definitions': {'dataset': {'properties': {'3d': {'description': 'Include if 3d dataset with value being the name of the 3rd dimension', 'type': 'string'}, 'file_path': {'description': 'Path to file used to create configuration.', 'type': 'string'}, 'name': {'description': 'Name of dataset', 'type': 'string'}, 'variables': {'description': 'Variables in file used to create configuration.', 'type': 'object'}}, 'required': ['name', 'file_path', 'variables'], 'title': 'Dataset description', 'type': 'object'}, 'variable': {'items': {'type': 'object'}, 'title': 'Variable map (variable name in QcConfig -> variable name in dataset', 'type': 'object'}}, 'description': 'Schema to validate configuration for QcCreatorConfig', 'properties': {'datasets': {'description': 'Array of datasets used for QcCreatorConfig', 'items': {'$ref': '#/definitions/dataset'}, 'type': 'array'}}, 'required': ['datasets'], 'title': 'QcConfigCreator Schema', 'type': 'object'})

Bases: dict

Defines the dataset(s) configuration used by QcConfigCreator.

Parameters:

• path_or_dict – QcConfigCreator configuration, one of the following formats: python dict JSON filepath (str or Path object)

• dict – JSON schema for CreatorConfig

class ioos_qc.config_creator.QcConfigCreator(creator_config)

Bases: object

Creates level-0 configuration to create QcQonfig.

Parameters:

creator_config (QcCreatorConfig) – Configuration for datasets and variables used to create qc_config.

allowed_stats

Specific statistics allowed to be used to configure each test.

Type:

list

allowed_operators

Operators allowed to used to configure each test.

Type:

list

_create_test_section(test_name, variable_config, test_limits)

Given test_name, QcVariableConfig and test_limits, return qc_config section for that test.

_determine_dataset_years()

Determine year used in datasets, return as dict {dataset_name, year}.

Notes

• Each dataset is from a unique climatology or source, so the monthly files have different years.

_get_stats(variable_config)

Return dict of stats (min, max, mean, std) for given config

_get_subset(var, bbox, time_slice, depth=0, pad_delta=0.5)

Get subset of data

_load_datasets()

Load datasets

_var2var_in_file(var)

Return variable name used in the dataset and dataset name

create_config(variable_config)

Create QARTOD QC config given QcVariableConfig.

Parameters:

variable_config (QcVariableConfig) – Config for variable to be quality controlled

Returns:

Config for ioos_qc

Return type:

qc_config (dict)

var2dataset(var)

Return dataset name and dataset for given variable (as named in qc_config, not in the file)

class ioos_qc.config_creator.QcVariableConfig(path_or_dict, schema={'definitions': {'test': {'description': 'Test (gross_range_test, etc.) definition', 'properties': {'fail_max': {'description': 'Function or value to define fail max', 'type': 'string'}, 'fail_min': {'description': 'Function or value to define fail min', 'type': 'string'}, 'suspect_max': {'description': 'Function or value to define suspect max', 'type': 'string'}, 'suspect_min': {'description': 'Function or value to define suspect min', 'type': 'string'}}, 'required': ['suspect_min', 'suspect_max', 'fail_min', 'fail_max'], 'type': 'object'}}, 'description': 'Schema to validate configuration for QcVariableConfig', 'properties': {'bbox': {'description': 'Bounding box of region in deployment (xmin, ymin, xmax, ymax)', 'type': 'array'}, 'end_time': {'description': 'Exclusive end time of deployment (YYYY-MM-DD)', 'type': 'string'}, 'start_time': {'description': 'Start time of deployment (YYYY-MM-DD)', 'type': 'string'}, 'tests': {'items': {'$ref': '#/definitions/test'}, 'type': 'object'}, 'variable': {'description': "Variable name. The same name used as a key in 'variable' object in CreatorConfig", 'type': 'string'}}, 'required': ['variable', 'bbox', 'start_time', 'end_time', 'tests'], 'title': 'QcVariableConfig', 'type': 'object'})

Bases: dict

Used to generate a QcConfig for a specific variable.

Parameters:

• path_or_dict – QcVariableConfig configuration, one of the following formats: python dict JSON filepath (str or Path object)

• schema – JSON schema for QcVariable

_validate_fx(input_fx, test_name)

Throws exception if input_fx contains tokens not specifically allowed

allowed_groupings = ['(', ')']

allowed_operators = ['+', '-', '*', '/']

allowed_stats = ['min', 'max', 'mean', 'std']

Submodules

ioos_qc.config_creator.config_creator module

class ioos_qc.config_creator.config_creator.CreatorConfig(path_or_dict, schema={'definitions': {'dataset': {'properties': {'3d': {'description': 'Include if 3d dataset with value being the name of the 3rd dimension', 'type': 'string'}, 'file_path': {'description': 'Path to file used to create configuration.', 'type': 'string'}, 'name': {'description': 'Name of dataset', 'type': 'string'}, 'variables': {'description': 'Variables in file used to create configuration.', 'type': 'object'}}, 'required': ['name', 'file_path', 'variables'], 'title': 'Dataset description', 'type': 'object'}, 'variable': {'items': {'type': 'object'}, 'title': 'Variable map (variable name in QcConfig -> variable name in dataset', 'type': 'object'}}, 'description': 'Schema to validate configuration for QcCreatorConfig', 'properties': {'datasets': {'description': 'Array of datasets used for QcCreatorConfig', 'items': {'$ref': '#/definitions/dataset'}, 'type': 'array'}}, 'required': ['datasets'], 'title': 'QcConfigCreator Schema', 'type': 'object'})

Bases: dict

Defines the dataset(s) configuration used by QcConfigCreator.

Parameters:

• path_or_dict – QcConfigCreator configuration, one of the following formats: python dict JSON filepath (str or Path object)

• dict – JSON schema for CreatorConfig

class ioos_qc.config_creator.config_creator.QcConfigCreator(creator_config)

Bases: object

Creates level-0 configuration to create QcQonfig.

Parameters:

creator_config (QcCreatorConfig) – Configuration for datasets and variables used to create qc_config.

allowed_stats

Specific statistics allowed to be used to configure each test.

Type:

list

allowed_operators

Operators allowed to used to configure each test.

Type:

list

_create_test_section(test_name, variable_config, test_limits)

Given test_name, QcVariableConfig and test_limits, return qc_config section for that test.

_determine_dataset_years()

Determine year used in datasets, return as dict {dataset_name, year}.

Notes

• Each dataset is from a unique climatology or source, so the monthly files have different years.

_get_stats(variable_config)

Return dict of stats (min, max, mean, std) for given config

_get_subset(var, bbox, time_slice, depth=0, pad_delta=0.5)

Get subset of data

_load_datasets()

Load datasets

_var2var_in_file(var)

Return variable name used in the dataset and dataset name

create_config(variable_config)

Create QARTOD QC config given QcVariableConfig.

Parameters:

variable_config (QcVariableConfig) – Config for variable to be quality controlled

Returns:

Config for ioos_qc

Return type:

qc_config (dict)

var2dataset(var)

Return dataset name and dataset for given variable (as named in qc_config, not in the file)

class ioos_qc.config_creator.config_creator.QcVariableConfig(path_or_dict, schema={'definitions': {'test': {'description': 'Test (gross_range_test, etc.) definition', 'properties': {'fail_max': {'description': 'Function or value to define fail max', 'type': 'string'}, 'fail_min': {'description': 'Function or value to define fail min', 'type': 'string'}, 'suspect_max': {'description': 'Function or value to define suspect max', 'type': 'string'}, 'suspect_min': {'description': 'Function or value to define suspect min', 'type': 'string'}}, 'required': ['suspect_min', 'suspect_max', 'fail_min', 'fail_max'], 'type': 'object'}}, 'description': 'Schema to validate configuration for QcVariableConfig', 'properties': {'bbox': {'description': 'Bounding box of region in deployment (xmin, ymin, xmax, ymax)', 'type': 'array'}, 'end_time': {'description': 'Exclusive end time of deployment (YYYY-MM-DD)', 'type': 'string'}, 'start_time': {'description': 'Start time of deployment (YYYY-MM-DD)', 'type': 'string'}, 'tests': {'items': {'$ref': '#/definitions/test'}, 'type': 'object'}, 'variable': {'description': "Variable name. The same name used as a key in 'variable' object in CreatorConfig", 'type': 'string'}}, 'required': ['variable', 'bbox', 'start_time', 'end_time', 'tests'], 'title': 'QcVariableConfig', 'type': 'object'})

Bases: dict

Used to generate a QcConfig for a specific variable.

Parameters:

• path_or_dict – QcVariableConfig configuration, one of the following formats: python dict JSON filepath (str or Path object)

• schema – JSON schema for QcVariable

_validate_fx(input_fx, test_name)

Throws exception if input_fx contains tokens not specifically allowed

allowed_groupings = ['(', ')']

allowed_operators = ['+', '-', '*', '/']

allowed_stats = ['min', 'max', 'mean', 'std']

ioos_qc.config_creator.config_creator.to_json(qc_config, out_file=None)

Given qc_config return json

ioos_qc.config_creator.fx_parser module

ioos_qc.config_creator.fx_parser.BNF()

expop :: ‘^’ multop :: ‘*’ | ‘/’ addop :: ‘+’ | ‘-’ integer :: [‘+’ | ‘-’] ‘0’..’9’+ atom :: PI | E | real | fn ‘(’ expr ‘)’ | ‘(’ expr ‘)’ factor :: atom [ expop factor ]* term :: factor [ multop factor ]* expr :: term [ addop term ]*

ioos_qc.config_creator.fx_parser.eval_fx(fx, stats)

Given fx and stats (‘min’, ‘max’, ‘mean’, ‘std’) return the result

ioos_qc.config_creator.fx_parser.evaluate_stack(s, stats)

ioos_qc.config_creator.fx_parser.push_first(toks)

ioos_qc.config_creator.fx_parser.push_unary_minus(toks)

ioos_qc.config_creator.get_assets module

Download and process source data used for ConfigCreator

ioos_qc.config_creator.get_assets.get_narr(output_dir)

ioos_qc.config_creator.get_assets.get_ocean_atlas(output_dir)

ioos_qc.config_creator.get_assets.main(output_dir, remove_tmp_files=False)

ioos_qc.config_creator.get_assets.narr_download(output_dir, sources={'NARR': {'url': 'ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.derived/surface/{0}'}, 'OCEAN_ATLAS': {'other_url': 'https://data.nodc.noaa.gov/thredds/fileServer/ncei/woa/{0}/all/1.00/woa18_all_{1}{2:02d}_01.nc', 'ts_url': 'https://data.nodc.noaa.gov/thredds/fileServer/ncei/woa/{0}/A5B7/1.00/woa18_A5B7_{1}{2:02d}_01.nc'}})

ioos_qc.config_creator.get_assets.narr_enhance(output_dir)

ioos_qc.config_creator.get_assets.narr_merge_variables(output_dir)

ioos_qc.config_creator.get_assets.ocean_atlas_download(output_dir, month, sources={'NARR': {'url': 'ftp://ftp.cdc.noaa.gov/Datasets/ncep.reanalysis.derived/surface/{0}'}, 'OCEAN_ATLAS': {'other_url': 'https://data.nodc.noaa.gov/thredds/fileServer/ncei/woa/{0}/all/1.00/woa18_all_{1}{2:02d}_01.nc', 'ts_url': 'https://data.nodc.noaa.gov/thredds/fileServer/ncei/woa/{0}/A5B7/1.00/woa18_A5B7_{1}{2:02d}_01.nc'}})

ioos_qc.config_creator.get_assets.ocean_atlas_enhance(output_dir)

ioos_qc.config_creator.get_assets.ocean_atlas_merge_time(output_dir)

ioos_qc.config_creator.get_assets.ocean_atlas_merge_variables(output_dir, month)

ioos_qc.config_creator.get_assets.ocean_atlas_variable_enhance(output_dir, month)

ioos_qc.config_creator.get_assets.remove_tmp_files(dirs_to_delete)

ioos_qc.config_creator.make_config module