onadata.libs.utils package

Submodules

onadata.libs.utils.api_export_tools module

API export util functions.

onadata.libs.utils.api_export_tools.custom_response_handler(request, xform, query, export_type, token=None, meta=None, dataview=False, filename=None)

Returns a HTTP response with export file for download.

onadata.libs.utils.api_export_tools.export_async_export_response(request, export)

Checks the export status and generates the reponse :param request: :param export: :return: response dict example {“job_status”: “Success”, “export_url”: …}

onadata.libs.utils.api_export_tools.external_export_response(export)

Redirects to export_url of XLSReports successful export. In case of a failure, returns a 400 HTTP JSON response with the error message.

onadata.libs.utils.api_export_tools.generate_google_web_flow(request)

Returns a OAuth2WebServerFlow object from the request redirect_uri.

onadata.libs.utils.api_export_tools.get_async_response(job_uuid, request, xform, count=0)

Returns the status of an async task for the given job_uuid.

onadata.libs.utils.api_export_tools.include_hxl_row(dv_columns, hxl_columns)

This function returns a boolean value. If the dataview’s columns are not part of the hxl columns, we return False. Returning False would mean that we don’t have to add the hxl column row if there aren’t any hxl columns in the dataview. :param dv_columns - dataview columns :param hxl_columns - hxl columns from the dataview’s xform

:return True or False

onadata.libs.utils.api_export_tools.log_export(request, xform, export_type)

Logs audit logs of export requests.

onadata.libs.utils.api_export_tools.process_async_export(request, xform, export_type, options=None)

Check if should generate export or just return the latest export. Rules for regenerating an export are:

  1. Filter included on the exports.

  2. New submission done.

  3. Always regenerate external exports.
    (External exports uses templates and the template might have

    changed)

Parameters

  • request –

  • xform –

  • export_type –

  • options – additional export params that may include query: export filter token: template url for xls external reports meta: metadataid that contains the external xls report template url remove_group_name: Flag to determine if group names should appear

Returns

response dictionary

onadata.libs.utils.api_export_tools.response_for_format(data, format=None)

Return appropriately formatted data in Response().

onadata.libs.utils.async_status module

onadata.libs.utils.async_status.async_status(status, error=None)
onadata.libs.utils.async_status.celery_state_to_status(state)

onadata.libs.utils.audit module

onadata.libs.utils.backup_tools module

onadata.libs.utils.backup_tools.create_zip_backup(zip_output_file, user, xform=None)
onadata.libs.utils.backup_tools.restore_backup_from_path(dir_path, username, status)

Only restores xml submissions, media files are assumed to still be in storage and will be retrieved by the filename stored within the submission

onadata.libs.utils.backup_tools.restore_backup_from_xml_file(xml_instance_path, username)
onadata.libs.utils.backup_tools.restore_backup_from_zip(zip_file_path, username)

onadata.libs.utils.briefcase_client module

class onadata.libs.utils.briefcase_client.BriefcaseClient(url, username, password, user)

Bases: object

download_instances(form_id, cursor=0, num_entries=100)
download_manifest(manifest_url, id_string)
download_media_files(xml_doc, media_path)
download_xforms(include_instances=False)
push()
onadata.libs.utils.briefcase_client.django_file(file_obj, field_name, content_type)
onadata.libs.utils.briefcase_client.node_value(node, tag_name)

Returns the first nodeValue of of an elementin the node with the matching tag_name otherwise returns empty list [].

onadata.libs.utils.cache_tools module

onadata.libs.utils.cache_tools.safe_delete(key)
onadata.libs.utils.cache_tools.safe_key(key)

Return a valid cache key

onadata.libs.utils.chart_tools module

onadata.libs.utils.chart_tools.build_chart_data(xform, language_index=0, page=0)
onadata.libs.utils.chart_tools.build_chart_data_for_field(xform, field, language_index=0, choices=None, group_by=None, data_view=None)
onadata.libs.utils.chart_tools.build_chart_data_from_widget(widget, language_index=0)
onadata.libs.utils.chart_tools.calculate_ranges(page, items_per_page, total_items)

Return the offset and end indices for a slice.

onadata.libs.utils.chart_tools.find_choice_label(choices, string)
onadata.libs.utils.chart_tools.get_chart_data_for_field(field_name, xform, accepted_format, group_by, field_xpath=None, data_view=None)

Get chart data for a given xlsform field.

onadata.libs.utils.chart_tools.get_choice_label(choices, string)

string is the name value found in the choices sheet.

Select one names should not contain spaces but some do and this conflicts with Select Multiple fields which use spaces to distinguish multiple choices.

A temporal fix to this is to search the choices list for both the full-string and the split keys.

onadata.libs.utils.chart_tools.get_field_choices(field, xform)

Retrieve field choices from a form survey element :param field: :param xform: :return: Form field choices

onadata.libs.utils.chart_tools.get_field_from_field_name(field_name, xform)
onadata.libs.utils.chart_tools.get_field_from_field_xpath(field_xpath, xform)
onadata.libs.utils.chart_tools.get_field_label(field, language_index=0)
onadata.libs.utils.chart_tools.utc_time_string_for_javascript(date_string)

Convert 2014-01-16T12:07:23.322+03 to 2014-01-16T12:07:23.322+03:00

Cant use datetime.str[fp]time here since python 2.7’s %z is platform dependant - http://stackoverflow.com/questions/2609259/converting-string-t o-datetime-object-in-python

onadata.libs.utils.common_tags module

onadata.libs.utils.common_tools module

Common helper functions

onadata.libs.utils.common_tools.cmp_to_key(mycmp)

Convert a cmp= function into a key= function

onadata.libs.utils.common_tools.filename_from_disposition(content_disposition)

Gets a filename from the given content disposition header.

onadata.libs.utils.common_tools.get_boolean_value(str_var, default=None)

Converts a string into boolean

onadata.libs.utils.common_tools.get_response_content(response, decode=True)

Gets HTTP content for the given a HTTP response object.

Handles the case where a streaming_content is in the response.

Parameters

  • response – The response to extract content from.

  • decode – If true decode as utf-8, default True.

onadata.libs.utils.common_tools.get_uuid()

Return UUID4 hex value

onadata.libs.utils.common_tools.json_stream(data, json_string)

Generator function to stream JSON data

onadata.libs.utils.common_tools.merge_dicts(*dict_args)

Given any number of dicts, shallow copy and merge into a new dict, precedence goes to key value pairs in latter dicts.

onadata.libs.utils.common_tools.report_exception(subject, info, exc_info=None)

Formats an exception then posts it to sentry and if not in debug or testing sends email to mail_admins.

onadata.libs.utils.common_tools.retry(tries, delay=3, backoff=2)
Adapted from code found here:

http://wiki.python.org/moin/PythonDecoratorLibrary#Retry

Retries a function or method until it returns True.

Parameters

delay – sets the initial delay in seconds, and backoff sets the

factor by which the delay should lengthen after each failure. :param backoff: must be greater than 1, or else it isn’t really a backoff. :param tries: must be at least 0, and delay greater than 0.

onadata.libs.utils.common_tools.str_to_bool(str_var)

Return boolean True or False if string s represents a boolean value

onadata.libs.utils.country_field module

class onadata.libs.utils.country_field.CountryField(*args, **kwargs)

Bases: django.db.models.fields.CharField

get_internal_type()

onadata.libs.utils.csv_builder module

class onadata.libs.utils.csv_builder.AbstractDataFrameBuilder(username, id_string, filter_query=None, group_delimiter='/', split_select_multiples=True, binary_select_multiples=False, start=None, end=None, remove_group_name=False, xform=None, include_labels=False, include_labels_only=False, include_images=True, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'), value_select_multiples=False, show_choice_labels=True, include_reviews=False, language=None, start_index=None, limit=None)

Bases: object

ADDITIONAL_COLUMNS = ['_id', '_uuid', '_submission_time', '_tags', '_notes', '_version', '_duration', '_submitted_by', '_total_media', '_media_count', '_media_all_received']
BINARY_SELECT_MULTIPLES = False
IGNORED_COLUMNS = ['_xform_id_string', '_status', '_attachments', '_geolocation', '_bamboo_dataset_id', '_deleted_at', '_edited']
VALUE_SELECT_MULTIPLES = False

Group functionality used by any DataFrameBuilder i.e. XLS, CSV and KML

class onadata.libs.utils.csv_builder.CSVDataFrameBuilder(username, id_string, filter_query=None, group_delimiter='/', split_select_multiples=True, binary_select_multiples=False, start=None, end=None, remove_group_name=False, xform=None, include_labels=False, include_labels_only=False, include_images=False, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'), value_select_multiples=False, show_choice_labels=False, include_reviews=False, language=None, csv_type='csv', start_index=None, limit=None)

Bases: onadata.libs.utils.csv_builder.AbstractDataFrameBuilder

comma_seperate_multi_choice_answer()
export_to(path, dataview=None)
onadata.libs.utils.csv_builder.get_column_names_only(columns, dd, group_delimiter)
onadata.libs.utils.csv_builder.get_end_name(columns, dd)

Return the name column for end entry

onadata.libs.utils.csv_builder.get_labels_from_columns(columns, dd, group_delimiter)
onadata.libs.utils.csv_builder.get_prefix_from_xpath(xpath)
onadata.libs.utils.csv_builder.ignore_start_stop(val)
onadata.libs.utils.csv_builder.old_write_to_csv(path, rows, columns, columns_with_hxl=None, remove_group_name=False, dd=None, group_delimiter='/', include_labels=False, include_labels_only=False, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'))
onadata.libs.utils.csv_builder.old_write_to_dhis2csv(username, id_string, path, rows, columns, dd=None, win_excel_utf8=False, total_records=None, orgunit_path=None, group_delimiter='/')

Writes export in dhis2 csv format to file value of rows can be very large and use up memory, hence, it should not be assigned to another variable else python with assign a memory location to it.

To generate the dhis2 csv,

  • get UID from get_dataelements function

and extracts the facility name to use as orgunits to populate the dataelement

and orgunit column respectively

onadata.libs.utils.csv_builder.remove_dups_from_list_maintain_order(l)
onadata.libs.utils.csv_builder.write_to_csv(path, rows, columns, columns_with_hxl=None, remove_group_name=False, dd=None, group_delimiter='/', include_labels=False, include_labels_only=False, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'))
onadata.libs.utils.csv_builder.write_to_dhis2csv(username, id_string, path, rows, columns, dd=None, win_excel_utf8=False, total_records=None, orgunit_path=None, group_delimiter='/')

Writes export in dhis2 csv format to file value of rows can be very large and use up memory, hence, it should not be assigned to another variable else python with assign a memory location to it.

To generate the dhis2 csv,

  • get UID from get_dataelements function

and extracts the facility name to use as orgunits to populate the dataelement

and orgunit column respectively

onadata.libs.utils.csv_import module

CSV data import module.

onadata.libs.utils.csv_import.dict2xmlsubmission(submission_dict, xform, instance_id, submission_date)

Creates and xml submission from an appropriate dict (& other data)

Parameters

  • submission_dict (dict) – A dict containing form submission data.

  • xfrom (onadata.apps.logger.models.XForm) – The submission’s XForm.

  • instance_id (string) – The submission/instance uuid.

  • submission_date (string) – An isoformatted datetime string.

Returns

An xml submission string

Return type

string

onadata.libs.utils.csv_import.dict_merge(a, b)

Returns a merger of two dicts a and b

credits: https://www.xormedia.com/recursively-merge-dictionaries-in-python

Parameters

  • a (dict) – The “Part A” dict

  • b (dict) – The “Part B” dict

Returns

The merger

Return type

dict

onadata.libs.utils.csv_import.dict_pathkeys_to_nested_dicts(dictionary)

Turns a flat dict to a nested dict

Takes a dict with pathkeys or “slash-namespaced” keys and inflates them into nested dictionaries i.e:- d[‘/path/key/123’] -> d[‘path’][‘key’][‘123’]

Parameters

dictionary (dict) – A dict with one or more “slash-namespaced” keys

Returns

A nested dict

Return type

dict

onadata.libs.utils.csv_import.failed_import(rollback_uuids, xform, exception, status_message)

Report a failed import. :param rollback_uuids: The rollback UUIDs :param xform: The XForm that failed to import to :param exception: The exception object :return: The async_status result

onadata.libs.utils.csv_import.get_async_csv_submission_status(job_uuid)

Gets CSV Submision progress or result Can be used to pol long running submissions :param str job_uuid: The submission job uuid returned by _submit_csv.delay :return: Dict with import progress info (insertions & total) :rtype: Dict

onadata.libs.utils.csv_import.get_submission_meta_dict(xform, instance_id)

Generates metadata for our submission

Checks if instance_id belongs to an existing submission. If it does, it’s considered an edit and its uuid gets deprecated. In either case, a new one is generated and assigned.

Parameters

  • xform (onadata.apps.logger.models.XForm) – The submission’s XForm.

  • instance_id (string) – The submission/instance uuid.

Returns

The metadata dict

Return type

dict

onadata.libs.utils.csv_import.submission_xls_to_csv(xls_file)

Convert a submission xls file to submissions csv file

Parameters

xls_file – submissions xls file

Returns

csv_file

onadata.libs.utils.csv_import.submit_csv(username, xform, csv_file, overwrite=False)

Imports CSV data to an existing form

Takes a csv formatted file or string containing rows of submission/instance and converts those to xml submissions and finally submits them by calling onadata.libs.utils.logger_tools.safe_create_instance()

Parameters

  • username (str) – the subission user

  • xfrom (onadata.apps.logger.models.XForm) – The submission’s XForm.

  • or file) ((str) – A CSV formatted file with submission rows.

Returns

If sucessful, a dict with import summary else dict with error str.

Return type

Dict

onadata.libs.utils.csv_reader module

class onadata.libs.utils.csv_reader.CsvReader(path)

Bases: object

Typical usage:

csv_reader = CsvReader(path)
for d in csv_reader.iter_dicts():
Variable.objects.create(**d)
close()
iter_dicts()
next()

A CsvReader object is iterable (since we have defined __iter__ and next methods. Each iteration of this object returns a row of data.

open(path)

onadata.libs.utils.decorators module

onadata.libs.utils.decorators.check_obj(f)
onadata.libs.utils.decorators.is_owner(view_func)

onadata.libs.utils.dict_tools module

Dict utility functions module.

onadata.libs.utils.dict_tools.csv_dict_to_nested_dict(csv_dict)

Converts a CSV dict to nested dicts.

onadata.libs.utils.dict_tools.dict_lists2strings(adict)

Convert lists in a dict to joined strings.

Parameters

d – The dict to convert.

Returns

The converted dict.

onadata.libs.utils.dict_tools.dict_paths2dict(adict)

Turns a dict with ‘/’ in keys to a nested dict.

onadata.libs.utils.dict_tools.floip_response_headers_dict(data, xform_headers)

Returns a dict from matching xform headers and floip responses.

onadata.libs.utils.dict_tools.get_values_matching_key(doc, key)

Returns iterator of values in ‘doc’ with the matching ‘key’.

onadata.libs.utils.dict_tools.list_to_dict(items, value)

Converts a list into a dict.

onadata.libs.utils.dict_tools.merge_list_of_dicts(list_of_dicts)

Merges a list of dicts to return one dict.

onadata.libs.utils.dict_tools.query_list_to_dict(query_list_str)

Returns a ‘label’ and ‘text’ from a Rapidpro values JSON string as a dict.

onadata.libs.utils.dict_tools.remove_indices_from_dict(obj)

Removes indices from a obj dict.

onadata.libs.utils.export_builder module

ExportBuilder

class onadata.libs.utils.export_builder.ExportBuilder

Bases: object

BINARY_SELECT_MULTIPLES = False
CONVERT_FUNCS = {'date': <function string_to_date_with_xls_validation>, 'dateTime': <function ExportBuilder.<lambda>>, 'decimal': <class 'float'>, 'int': <class 'int'>}
EXTRA_FIELDS = ['_id', '_uuid', '_submission_time', '_index', '_parent_table_name', '_parent_index', '_tags', '_notes', '_version', '_duration', '_submitted_by']
GROUP_DELIMITER = '/'
GROUP_DELIMITERS = ['/', '.']
GROUP_DELIMITER_DOT = '.'
GROUP_DELIMITER_SLASH = '/'
IGNORED_COLUMNS = ['_xform_id_string', '_status', '_attachments', '_geolocation', '_bamboo_dataset_id', '_deleted_at']
INCLUDE_HXL = False
INCLUDE_IMAGES = True
INCLUDE_LABELS = False
INCLUDE_LABELS_ONLY = False
INCLUDE_REVIEWS = False
REPEAT_INDEX_TAGS = ('[', ']')
SHOW_CHOICE_LABELS = False
SPLIT_SELECT_MULTIPLES = True
TRUNCATE_GROUP_TITLE = False
TYPES_TO_CONVERT = ['int', 'decimal', 'date']
VALUE_SELECT_MULTIPLES = False
XLS_SHEET_NAME_MAX_CHARS = 31
classmethod convert_type(value, data_type)

Convert data to its native type e.g. string ‘1’ to int 1 @param value: the string value to convert @param data_type: the native data type to convert to @return: the converted value

classmethod decode_mongo_encoded_fields(row, encoded_fields)
classmethod format_field_title(abbreviated_xpath, field_delimiter, data_dictionary, remove_group_name=False)
get_choice_label_from_dict(label)
get_default_language(languages)
get_fields(dataview, section, key)

Return list of element value with the key in section[‘elements’].

classmethod get_valid_sheet_name(desired_name, existing_names)
language = None
pre_process_row(row, section)

Split select multiples, gps and decode . and $

section_by_name(name)
set_survey(survey, xform=None, include_reviews=False)
classmethod split_gps_components(row, gps_fields)
classmethod split_select_multiples(row, select_multiples, select_values=False, binary_select_multiples=False, show_choice_labels=False, data_dictionary=None, language=None)

Split select multiple choices in a submission to individual columns.

Parameters

  • row – single submission dict

  • select_multiples – list of XPATHs and choices of select multiple questions.

  • binary_select_multiples – if True the value of the split columns will be 1 when the choice has been selected otherwise it will be 0.

  • select_values – the value of the split columns will be the name/value of the choice when selected otherwise blank/None.

  • show_choice_labels – Show a choice label instead of the value/True/False/1/0.

  • data_dictionary – A DataDictionary/XForm object

  • language – specific language as defined in the XLSForm.

Returns

the row dict with select multiples choice as fields in the row

to_csv(path, data, username, id_string, filter_query, **kwargs)

Generates a CSV file for submitted data based on the csv_type.

to_dhis2csv_export(path, data, username, id_string, filter_query, **kwargs)

Generates a DHIS2CSV file for submitted data.

to_flat_csv_export(path, data, username, id_string, filter_query, **kwargs)

Generates a flattened CSV file for submitted data.

to_xls_export(path, data, *args, **kwargs)
to_zipped_csv(path, data, *args, **kwargs)
to_zipped_sav(path, data, *args, **kwargs)
url = None
onadata.libs.utils.export_builder.current_site_url(path)

Returns fully qualified URL (no trailing slash) for the current site. :param path :return: complete url

onadata.libs.utils.export_builder.decode_mongo_encoded_section_names(data)

Recursively decode mongo keys.

Parameters

data – A dictionary to decode.

onadata.libs.utils.export_builder.dict_to_joined_export(data, index, indices, name, survey, row, media_xpaths=[])

Converts a dict into one or more tabular datasets :param data: current record which can be changed or updated :param index: keeps count of record number :param indices: a dictionary storing list values if data is a dict :param name: the name of the survey :param survey: the survey :param row: current record that remains unchanged on this function’s recall

onadata.libs.utils.export_builder.encode_if_str(row, key, encode_dates=False, sav_writer=None)
onadata.libs.utils.export_builder.get_choice_label(label, data_dictionary, language=None)

Return the label matching selected language or simply just the label.

onadata.libs.utils.export_builder.get_choice_label_value(key, value, data_dictionary, language=None)

Return the label of a choice matching the value if the key xpath is a SELECT_ONE otherwise it returns the value unchanged.

onadata.libs.utils.export_builder.get_data_dictionary_from_survey(survey)
onadata.libs.utils.export_builder.get_value_or_attachment_uri(key, value, row, data_dictionary, media_xpaths, attachment_list=None, show_choice_labels=False, language=None)

Gets either the attachment value or the attachment url :param key: used to retrieve survey element :param value: filename :param row: current records row :param data_dictionary: form structure :param include_images: boolean value to either inlcude images or not :param attachment_list: to be used incase row doesn’t have ATTACHMENTS key :return: value

onadata.libs.utils.export_builder.is_all_numeric(items)

Check if all items on the list are numeric, zero padded numbers will not be considered as numeric.

Parameters

items – list of values to be checked

Return boolean

onadata.libs.utils.export_builder.string_to_date_with_xls_validation(date_str)

Try to convert a string to a date object.

Parameters

date_str – string to convert

Returns

object if converted, otherwise date string

onadata.libs.utils.export_builder.track_task_progress(additions, total=None)

Updates the current export task with number of submission processed. Updates in batches of settings EXPORT_TASK_PROGRESS_UPDATE_BATCH defaults to 100. :param additions: :param total: :return:

onadata.libs.utils.export_tools module

Export tools

onadata.libs.utils.export_tools.check_pending_export(xform, export_type, options, minutes=5)

Check for pending export done within a specific period of time and returns the export :param xform: :param export_type: :param options: :param minutes :return:

onadata.libs.utils.export_tools.clean_keys_of_slashes(record)

Replaces the slashes found in a dataset keys with underscores :param record: list containing a couple of dictionaries :return: record with keys without slashes

onadata.libs.utils.export_tools.create_export_object(xform, export_type, options)

Return an export object that has not been saved to the database.

onadata.libs.utils.export_tools.generate_attachments_zip_export(export_type, username, id_string, export_id=None, options=None, xform=None)

Generates zip export of attachments.

param: export_type params: username: logged in username params: id_string: xform id_string params: export_id: ID of export object associated with the request param: options: additional parameters required for the lookup.

ext: File extension of the generated export

onadata.libs.utils.export_tools.generate_external_export(export_type, username, id_string, export_id=None, options=None, xform=None)

Generates external export using ONA data through an external service.

param: export_type params: username: logged in username params: id_string: xform id_string params: export_id: ID of export object associated with the request param: options: additional parameters required for the lookup.

data_id: instance id query: filter_query for custom queries meta: metadata associated with external export token: authentication key required by external service

onadata.libs.utils.export_tools.generate_kml_export(export_type, username, id_string, export_id=None, options=None, xform=None)

Generates kml export for geographical data

Parameters

  • export_type – type of export

  • username – logged in username

  • id_string – xform id_string

  • export_id – ID of export object associated with the request

  • options – additional parameters required for the lookup.

  • extension – File extension of the generated export

onadata.libs.utils.export_tools.generate_osm_export(export_type, username, id_string, export_id=None, options=None, xform=None)

Generates osm export for OpenStreetMap data

Parameters

  • export_type – type of export

  • username – logged in username

  • id_string – xform id_string

  • export_id – ID of export object associated with the request

  • options – additional parameters required for the lookup.

  • ext – File extension of the generated export

onadata.libs.utils.export_tools.get_export_options(options)

Returns expirt options as defined in Export.EXPORT_OPTION_FIELDS from a list of provided options to be saved with each Export object.

onadata.libs.utils.export_tools.get_or_create_export(export_id, xform, export_type, options)

Returns an existing export object or creates a new one with the given options.

onadata.libs.utils.export_tools.get_or_create_export_object(export_id, options, xform, export_type)

Get or create export object.

Parameters

  • export_id – Export ID

  • options – Options to convert to export options

  • xform – XForm to export

  • export_type – The type of export

Returns

A new or found export object

onadata.libs.utils.export_tools.get_osm_data_kwargs(xform)

Return kwargs for OsmData queryset for given xform

onadata.libs.utils.export_tools.get_repeat_index_tags(index_tags)

Gets a comma separated string index_tags

Retuns a tuple of two strings with SUPPORTED_INDEX_TAGS,

onadata.libs.utils.export_tools.increment_index_in_filename(filename)

filename should be in the form file.ext or file-2.ext - we check for the dash and index and increment appropriately

onadata.libs.utils.export_tools.kml_export_data(id_string, user, xform=None)

KML export data from form submissions.

onadata.libs.utils.export_tools.md5hash(string)

Return the MD5 hex digest of the given string.

onadata.libs.utils.export_tools.newest_export_for(xform, export_type, options)

Retrieve the latest export given the following arguments:

param: xform param: export_type param: options: additional parameters required for the lookup.

remove_group_name: boolean flag group_delimiter: “/” or “.” with “/” as the default split_select_multiples: boolean flag binary_select_multiples: boolean flag index_tag: (‘[‘, ‘]’) or (‘_’, ‘_’)

onadata.libs.utils.export_tools.parse_request_export_options(params)

Parse export options in the request object into values returned in a list. The list represents a boolean for whether the group name should be removed, the group delimiter, and a boolean for whether select multiples should be split.

onadata.libs.utils.export_tools.should_create_new_export(xform, export_type, options, request=None)

Function that determines whether to create a new export. param: xform param: export_type param: options: additional parameters required for the lookup.

remove_group_name: boolean flag group_delimiter: “/” or “.” with “/” as the default split_select_multiples: boolean flag binary_select_multiples: boolean flag index_tag: (‘[‘, ‘]’) or (‘_’, ‘_’)

params: request: Get params are used to determine if new export is required

onadata.libs.utils.export_tools.upload_template_for_external_export(server, file_obj)

Uploads an Excel template to the XLSReport server.

Returns the status code with the server response.

onadata.libs.utils.export_tools.write_temp_file_to_path(suffix, content, file_path)

Write a temp file and return the name of the file. :param suffix: The file suffix :param content: The content to write :param file_path: The path to write the temp file to :return: The filename written to

onadata.libs.utils.google module

onadata.libs.utils.gravatar module

onadata.libs.utils.gravatar.email_md5(user)
onadata.libs.utils.gravatar.gravatar_exists(user)

onadata.libs.utils.image_tools module

onadata.libs.utils.image_tools.flat(*nums)

Build a tuple of ints from float or integer arguments. Useful because PIL crop and resize require integer points. source: https://gist.github.com/16a01455

onadata.libs.utils.image_tools.get_dimensions(size, longest_side)
onadata.libs.utils.image_tools.image_url(attachment, suffix)

Return url of an image given size(@param suffix) e.g large, medium, small, or generate required thumbnail

onadata.libs.utils.image_tools.resize(filename, extension)
onadata.libs.utils.image_tools.resize_local_env(filename, extension)

onadata.libs.utils.log module

class onadata.libs.utils.log.AuditLogHandler(model='')

Bases: logging.Handler

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

get_model(name)
class onadata.libs.utils.log.Enum(**enums)

Bases: object

onadata.libs.utils.log.audit_log(action, request_user, account_user, message, audit, request, level=10)

Create a log message based on these params

@param action: Action performed e.g. form-deleted @param request_username: User performing the action @param account_username: The formhub account the action was performed on @param message: The message to be displayed on the log @param level: log level @param audit: a dict of key/values of other info pertaining to the action e.g. form’s id_string, submission uuid @return: None

onadata.libs.utils.logger_tools module

class onadata.libs.utils.logger_tools.BaseOpenRosaResponse(*args, **kwargs)

Bases: django.http.response.HttpResponse

status_code = 201
class onadata.libs.utils.logger_tools.OpenRosaResponse(*args, **kwargs)

Bases: onadata.libs.utils.logger_tools.BaseOpenRosaResponse

status_code = 201
class onadata.libs.utils.logger_tools.OpenRosaResponseBadRequest(*args, **kwargs)

Bases: onadata.libs.utils.logger_tools.OpenRosaResponse

status_code = 400
class onadata.libs.utils.logger_tools.OpenRosaResponseForbidden(*args, **kwargs)

Bases: onadata.libs.utils.logger_tools.OpenRosaResponse

status_code = 403
class onadata.libs.utils.logger_tools.OpenRosaResponseNotAllowed(*args, **kwargs)

Bases: onadata.libs.utils.logger_tools.OpenRosaResponse

status_code = 405
class onadata.libs.utils.logger_tools.OpenRosaResponseNotFound(*args, **kwargs)

Bases: onadata.libs.utils.logger_tools.OpenRosaResponse

status_code = 404
class onadata.libs.utils.logger_tools.PublishXForm(xml_file, user)

Bases: object

publish_xform()
onadata.libs.utils.logger_tools.check_edit_submission_permissions(request_user, xform)
onadata.libs.utils.logger_tools.check_submission_permissions(request, xform)

Check that permission is required and the request user has permission.

The user does no have permissions iff:

  • the user is authed,

  • either the profile or the form require auth,

  • the xform user is not submitting.

Since we have a username, the Instance creation logic will handle checking for the forms existence by its id_string.

Returns

None.

Raises

PermissionDenied based on the above criteria.

onadata.libs.utils.logger_tools.create_instance(username, xml_file, media_files, status='submitted_via_web', uuid=None, date_created_override=None, request=None)

I used to check if this file had been submitted already, I’ve taken this out because it was too slow. Now we’re going to create a way for an admin to mark duplicate instances. This should simplify things a bit. Submission cases: * If there is a username and no uuid, submitting an old ODK form. * If there is a username and a uuid, submitting a new ODK form.

onadata.libs.utils.logger_tools.dict2xform(jsform, form_id, root=None)
onadata.libs.utils.logger_tools.generate_content_disposition_header(name, extension, show_date=True)
onadata.libs.utils.logger_tools.get_filtered_instances(*args, **kwargs)

Get filtered instances - mainly to allow mocking in tests

onadata.libs.utils.logger_tools.get_first_record(queryset)

Returns the first item in a queryset sorted by id.

onadata.libs.utils.logger_tools.get_uuid_from_submission(xml)
onadata.libs.utils.logger_tools.get_xform_from_submission(xml, username, uuid=None)
onadata.libs.utils.logger_tools.inject_instanceid(xml_str, uuid)
onadata.libs.utils.logger_tools.publish_form(callback)

Calls the callback function to publish a XLSForm and returns appropriate message depending on exception throw during publishing of a XLSForm.

onadata.libs.utils.logger_tools.publish_xls_form(xls_file, user, project, id_string=None, created_by=None)

Create or update DataDictionary with xls_file, user id_string is optional when updating

onadata.libs.utils.logger_tools.publish_xml_form(xml_file, user, project, id_string=None, created_by=None)
onadata.libs.utils.logger_tools.remove_xform(xform)
onadata.libs.utils.logger_tools.response_with_mimetype_and_name(mimetype, name, extension=None, show_date=True, file_path=None, use_local_filesystem=False, full_mime=False)
onadata.libs.utils.logger_tools.safe_create_instance(username, xml_file, media_files, uuid, request)

Create an instance and catch exceptions.

Returns

A list [error, instance] where error is None if there was no error.

onadata.libs.utils.logger_tools.save_attachments(xform, instance, media_files)

Saves attachments for the given instance/submission.

onadata.libs.utils.logger_tools.save_submission(xform, xml, media_files, new_uuid, submitted_by, status, date_created_override, checksum)
onadata.libs.utils.logger_tools.store_temp_file(data)
onadata.libs.utils.logger_tools.update_attachment_tracking(instance)

Takes an Instance object and updates attachment tracking fields

onadata.libs.utils.middleware module

class onadata.libs.utils.middleware.BaseMiddleware(get_response)

Bases: object

class onadata.libs.utils.middleware.ExceptionLoggingMiddleware(get_response)

Bases: object

process_exception(request, exception)
class onadata.libs.utils.middleware.HTTPResponseNotAllowedMiddleware(get_response)

Bases: object

class onadata.libs.utils.middleware.LocaleMiddlewareWithTweaks(get_response=None)

Bases: django.middleware.locale.LocaleMiddleware

Overrides LocaleMiddleware from django with:

Khmer km language code in Accept-Language is rewritten to km-kh

process_request(request)
class onadata.libs.utils.middleware.OperationalErrorMiddleware(get_response)

Bases: onadata.libs.utils.middleware.BaseMiddleware

Captures requests returning 500 status code. Then retry it against master database.

process_exception(request, exception)
class onadata.libs.utils.middleware.SqlLogging(get_response)

Bases: object

onadata.libs.utils.model_tools module

Model utility functions.

onadata.libs.utils.model_tools.get_columns_with_hxl(survey_elements)

Returns a dictionary whose keys are xform field names and values are instance::hxl values set on the xform :param include_hxl - boolean value :param survey_elements - survey elements of an xform return dictionary or None

onadata.libs.utils.model_tools.queryset_iterator(queryset, chunksize=100)

Iterate over a Django Queryset.

This method loads a maximum of chunksize (default: 100) rows in its memory at the same time while django normally would load all rows in its memory. Using the iterator() method only causes it to not preload all the classes.

See https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator

onadata.libs.utils.model_tools.set_uuid(obj)

Only give an object a new UUID if it does not have one.

onadata.libs.utils.mongo module

onadata.libs.utils.numeric module

onadata.libs.utils.numeric.int_or_parse_error(value, error_string)

If value is not an int raise a parse error with error_string, which is a format string that takes one argument, the value.

onadata.libs.utils.osm module

OSM utility module.

onadata.libs.utils.osm.get_combined_osm(osm_list)

Combine osm xml form list of OsmData objects

onadata.libs.utils.osm.osm_flat_dict(instance_id)

Flat dict of OSM tags for the specified submission.

Each key starts with ‘osm_*’.

onadata.libs.utils.osm.parse_osm(osm_xml, include_osm_id=False)

Parses OSM XML and return a list of ways or nodes.

onadata.libs.utils.osm.parse_osm_nodes(osm_xml, include_osm_id=False)

Converts an OSM XMl to a list of GEOSGeometry objects

onadata.libs.utils.osm.parse_osm_tags(node, include_osm_id=False)

Retrieves all the tags from a osm xml node

onadata.libs.utils.osm.parse_osm_ways(osm_xml, include_osm_id=False)

Converts an OSM XMl to a list of GEOSGeometry objects

onadata.libs.utils.osm.save_osm_data(instance_id)

Includes the OSM data in the specified submission json data.

onadata.libs.utils.profiler module

onadata.libs.utils.project_utils module

project_utils module - apply project permissions to a form.

onadata.libs.utils.project_utils.set_project_perms_to_xform(xform, project)

Apply project permissions to a form, this usually happens when a new form is being published or it is being moved to a new project.

onadata.libs.utils.qrcode module

onadata.libs.utils.qrcode.generate_qrcode(message, stream=None, eclevel='M', margin=10, data_mode='8bits', format='PNG', scale=2.5)

Generate a QRCode, settings options and output.

onadata.libs.utils.quick_converter module

class onadata.libs.utils.quick_converter.QuickConverter(data=None, files=None, auto_id='id_%s', prefix=None, initial=None, error_class=<class 'django.forms.utils.ErrorList'>, label_suffix=None, empty_permitted=False, field_order=None, use_required_attribute=None, renderer=None)

Bases: django.forms.forms.Form

base_fields = {'xls_file': <django.forms.fields.FileField object>}
declared_fields = {'xls_file': <django.forms.fields.FileField object>}
media
publish(user)

onadata.libs.utils.string module

onadata.libs.utils.string.str2bool(v)

onadata.libs.utils.timing module

onadata.libs.utils.timing.calculate_duration(start_time, end_time)

This function calculates duration when given start and end times. An empty string is returned if either of the time formats does not match ‘_format’ format else, the duration is returned

onadata.libs.utils.timing.get_date(_object=None)
onadata.libs.utils.timing.get_header_date_format(date_modified)
onadata.libs.utils.timing.last_modified_header(last_modified_date)

onadata.libs.utils.user_auth module

class onadata.libs.utils.user_auth.HttpResponseNotAuthorized

Bases: django.http.response.HttpResponse

status_code = 401
onadata.libs.utils.user_auth.add_cors_headers(response)
onadata.libs.utils.user_auth.basic_http_auth(func)
onadata.libs.utils.user_auth.check_and_set_form_by_id(pk, request)
onadata.libs.utils.user_auth.check_and_set_form_by_id_string(username, id_string, request)
onadata.libs.utils.user_auth.check_and_set_user(request, username)
onadata.libs.utils.user_auth.check_and_set_user_and_form(username, id_string, request)
onadata.libs.utils.user_auth.get_user_default_project(user)
onadata.libs.utils.user_auth.get_xform_and_perms(username, id_string, request)
onadata.libs.utils.user_auth.get_xform_users_with_perms(xform)

Similar to django-guadian’s get_users_with_perms here the query makes use of the xformuserobjectpermission_set to return a dictionary of users with a list of permissions to the XForm object. The query in use is not as expensive as the one in use with get_users_with_perms

onadata.libs.utils.user_auth.has_edit_permission(xform, owner, request, shared=False)
onadata.libs.utils.user_auth.has_permission(xform, owner, request, shared=False)
onadata.libs.utils.user_auth.has_submit_permission(xform, owner, request, shared=False)
onadata.libs.utils.user_auth.helper_auth_helper(request)
onadata.libs.utils.user_auth.http_auth_string(username, password)
onadata.libs.utils.user_auth.set_api_permissions_for_user(user)
onadata.libs.utils.user_auth.set_profile_data(data, content_user)

onadata.libs.utils.viewer_tools module

Util functions for data views.

onadata.libs.utils.viewer_tools.create_attachments_zipfile(attachments)

Return a zip file with submission attachments.

onadata.libs.utils.viewer_tools.django_file(path, field_name, content_type)

Return an InMemoryUploadedFile object for file uploads.

onadata.libs.utils.viewer_tools.enketo_url(form_url, id_string, instance_xml=None, instance_id=None, return_url=None, **kwargs)

Return Enketo webform URL.

onadata.libs.utils.viewer_tools.export_def_from_filename(filename)

Return file extension and mimetype from filename.

onadata.libs.utils.viewer_tools.generate_enketo_form_defaults(xform, **kwargs)

Return Enketo default options for preloading data into a web form.

onadata.libs.utils.viewer_tools.get_client_ip(request)

Return an IP from HTTP_X_FORWARDED_FOR or REMOTE_ADDR request headers.

arguments: request – HttpRequest object.

onadata.libs.utils.viewer_tools.get_enketo_edit_url(request, instance, return_url)

Given a submssion instance, returns an Enketo link to edit the specified submission.

onadata.libs.utils.viewer_tools.get_enketo_preview_url(request, username, id_string, xform_pk=None)

Return an Enketo preview URL.

onadata.libs.utils.viewer_tools.get_enketo_single_submit_url(request, username, id_string, xform_pk=None)

Return single submit url of the submission instance.

onadata.libs.utils.viewer_tools.get_form(kwargs)

Return XForm object by applying kwargs on an XForm queryset.

onadata.libs.utils.viewer_tools.get_form_url(request, username=None, protocol='https', preview=False, xform_pk=None)

Return a form list url endpoint to be used to make a request to Enketo.

For example, it will return https://example.com and Enketo will know to look for the form list at https://example.com/formList. If a username is provided then Enketo will request the form list from https://example.com/[username]/formList. Same applies for preview if preview is True and also to a single form when xform_pk is provided.

onadata.libs.utils.viewer_tools.get_path(path, suffix)

Apply the suffix to the path.

onadata.libs.utils.viewer_tools.handle_enketo_error(response)

Handle enketo error response.

onadata.libs.utils.viewer_tools.image_urls(instance)

Return image urls of all image attachments of the submission instance.

arguments: instance – Instance submission object.

onadata.libs.utils.viewer_tools.image_urls_for_form(xform)

Return image urls of all image attachments of the xform.

onadata.libs.utils.viewer_tools.parse_xform_instance(xml_str)

‘xml_str’ is a str object holding the XML of an XForm instance.

Return a python object representation of this XML file.

Module contents