Module docplex.util.environment

Representation of the DOcplex solving environment.

This module handles the various elements that allow an optimization program to run independently from the solving environment. This environment may be:

• on premise, using a local version of CPLEX Optimization Studio to solve MP problems, or
• on Watson Machine Learning, with the Python program running inside the Python Worker.
• on Decision Optimization in Watson Machine Learning.

As much as possible, the adaptation to the solving environment is automatic. The functions that are presented here are useful for handling very specific use cases.

The following code is a program that sums its input (sum.py):

import json
import docplex.util.environment as environment

sum = 0
# open program input named "data.txt" and sum the contents
with environment.get_input_stream("data.txt") as input:
    for i in input.read().split():
        sum += int(i)
# write the result as a simple json in program output "solution.json"
with environment.get_output_stream("solution.json") as output:
    output.write(json.dumps({'result': sum}))

Let’s put some data in a data.txt file:

4 7 8
19

When you run sum.py with a Python interpreter, it opens the data.txt file and sums all of the integers in that file. The result is saved as a JSON fragment in file solution.json:

$ python sum.py
$ more solution.json
{"result": 38}

Environment representation can be accessed with different ways:

• direct object method calls, after retrieving an instance using docplex.util.environment.get_environment() and using methods of docplex.util.environment.Environment.

• using the function in package docplex.util.environment. They will call

  the corresponding methods of Environment in the platform default_environment:

  • docplex.util.environment.get_input_stream()
  • docplex.util.environment.get_output_stream()
  • docplex.util.environment.read_df()
  • docplex.util.environment.write_df()
  • docplex.util.environment.get_available_core_count()
  • docplex.util.environment.get_parameter()
  • docplex.util.environment.update_solve_details()
  • docplex.util.environment.add_abort_callback()
  • docplex.util.environment.remove_abort_callback()

class docplex.util.environment.AbstractLocalEnvironment

Bases: docplex.util.environment.Environment

get_available_core_count()

Returns the number of cores available for processing if the environment sets a limit.

This number is used in the solving engine as the number of threads.

Returns:The available number of cores or None if the environment does not limit the number of cores.

get_input_stream(name)

Get an input of the program as a stream (file-like object).

An input of the program is a file that is available in the working directory.

When run on Watson Machine Learning, all input attachments are copied to the working directory before the program is run. get_input_stream lets you open the input attachments of the job.

Parameters:name – Name of the input object. Returns:A file object to read the input from.

get_output_stream(name)

Get a file-like object to write the output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, the output of the program is written as files in the working directory. When run on Watson Machine Learning, the files are attached as output attachments.

The stream is opened in binary mode, and will accept 8 bits data.

Parameters:name – Name of the output object. Returns:A file object to write the output to.

get_parameter(name)

Returns a parameter of the program.

on Watson Machine Learning, this method returns the job parameter whose name is specified. On local solver, this method returns the environment variable whose name is specified.

Parameters:name – The name of the parameter. Returns:The parameter whose name is specified or None if the parameter does not exists.

get_parameters()

Returns a dict containing all parameters of the program.

on Watson Machine Learning, this method returns the job parameters. On local solver, this method returns os.environ.

Returns:The job parameters

set_output_attachment(name, filename)

Attach the file which filename is specified as an output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, filename is copied to the the working directory (if not already there) under the name name.

When run on Watson Machine Learning, the file is attached as output attachment.

Parameters:

• name – Name of the output object.
• filename – The name of the file to attach.

class docplex.util.environment.Environment

Bases: object

Methods for interacting with the execution environment.

Internally, the docplex package provides the appropriate implementation according to the actual execution environment. The correct instance of this class is returned by the method docplex.util.environment.get_environment() that is provided in this module.

abort_callbacks

A list of callbacks that are called when the script is run on Watson Machine Learning and a job abort operation is requested. You add your own callback using:

env.abort_callbacks += [your_cb]

or:

env.abort_callbacks.append(your_cb)

You remove a callback using:

env.abort_callbacks.remove(your_cb)

solution_storage_handler

A function called when a solution is to be stored. The storage handler is a function which first argument is the Environment on which a solution should be saved. The solution is a dict containing all the data for an optimization solution. The default is default_solution_storage_handler().

record_history_fields

Fields which history is to be kept

record_history_size

maximum number of records in history

record_interval

min time between to history records

get_available_core_count()

Returns the number of cores available for processing if the environment sets a limit.

This number is used in the solving engine as the number of threads.

Returns:The available number of cores or None if the environment does not limit the number of cores.

get_engine_log_level()

Returns the engine log level as set by job parameter oaas.engineLogLevel.

oaas.engineLogLevel values are: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL

The mapping to logging levels in python are:

• OFF: None
• SEVERE: logging.ERROR
• WARNING: logging.WARNING
• INFO, CONFIG: logging.INFO
• FINE, FINER, FINEST: logging.DEBUG
• ALL: logging.DEBUG

All other values are considered invalid values and will return None.

Returns:The logging level or None if not set (off)

get_input_stream(name)

Get an input of the program as a stream (file-like object).

An input of the program is a file that is available in the working directory.

When run on Watson Machine Learning, all input attachments are copied to the working directory before the program is run. get_input_stream lets you open the input attachments of the job.

Parameters:name – Name of the input object. Returns:A file object to read the input from.

get_output_stream(name)

Get a file-like object to write the output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, the output of the program is written as files in the working directory. When run on Watson Machine Learning, the files are attached as output attachments.

The stream is opened in binary mode, and will accept 8 bits data.

Parameters:name – Name of the output object. Returns:A file object to write the output to.

get_parameter(name)

Returns a parameter of the program.

on Watson Machine Learning, this method returns the job parameter whose name is specified. On local solver, this method returns the environment variable whose name is specified.

Parameters:name – The name of the parameter. Returns:The parameter whose name is specified or None if the parameter does not exists.

get_parameters()

Returns a dict containing all parameters of the program.

on Watson Machine Learning, this method returns the job parameters. On local solver, this method returns os.environ.

Returns:The job parameters

get_stop_callback()

Returns the stop callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.

You can also use the stop_callback property to get the callback.

Deprecated since 2.4 - Use the abort_callbacks property instead’)

is_debug_mode()

Returns true if the engine should run in debug mode.

This is equivalent to env.get_engine_log_level() <= logging.DEBUG

publish_solve_details(details)

Actually publish the solve specified details.

Returns:The published details

read_df(name, reader=None, **kwargs)

Reads an input of the program as a pandas.DataFrame.

pandas must be installed.

name is the name of the input object, as a filename. If a reader is not user provided, the reader used depends on the filename extension.

The default reader used depending on extension are:

• .csv: pandas.read_csv()
• .msg: pandas.read_msgpack()

Parameters:

• name – The name of the input object
• reader – an optional reader function
• **kwargs – additional parameters passed to the reader

Raises:

NotAvailableError – raises this error when pandas is not available.

set_output_attachment(name, filename)

Attach the file which filename is specified as an output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, filename is copied to the the working directory (if not already there) under the name name.

When run on Watson Machine Learning, the file is attached as output attachment.

Parameters:

• name – Name of the output object.
• filename – The name of the file to attach.

set_stop_callback(cb)

Sets a callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.

You can also use the stop_callback property to set the callback.

Deprecated since 2.4 - Use self.abort_callbacks += [cb] instead’

Parameters:cb – The callback function

stop_callback

Returns the stop callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.

You can also use the stop_callback property to get the callback.

Deprecated since 2.4 - Use the abort_callbacks property instead’)

store_solution(solution)

Stores the specified solution.

This method guarantees that the solution is fully saved if the model is running on Watson Machine Learning python worker and an abort of the job is triggered.

For each (key, value) pairs of the solution, the default solution storage handler does the following depending of the type of value, in the following order:

• If value is a pandas.DataFrame, then the data frame is saved as an output with the specified name. Note that name must include an extension file for the serialization. See Environment.write_df() for supported formats.
• If value is a bytes, it is saved as binary data with name.
• The value is saved as an output with the name, after it has been converted to JSON.

Parameters:solution – a dict containing the solution.

update_solve_details(details)

Update the solve details.

You use this method to send solve details to the solve service. If context.solver.auto_publish is set, the underlying solver will automatically update solve details once the solve has finished.

This method might filter details and publish them with rate limitations. It actually publish de details by calling publish_solve_details.

Parameters:details – A dict with solve details as key/value pairs.

write_df(df, name, writer=None, **kwargs)

Write a pandas.DataFrame as an output of the program.

pandas must be installed.

name is the name of the input object, as a filename. If a writer is not user provided, the writer used depends on the filename extension.

This currently only supports csv output.

Parameters:

• name – The name of the input object
• writer – an optional writer function
• **kwargs – additional parameters passed to the writer

Raises:

NotAvailableError – raises this error when pandas is not available.

class docplex.util.environment.LocalEnvironment

Bases: docplex.util.environment.AbstractLocalEnvironment

exception docplex.util.environment.NotAvailableError

Bases: Exception

The exception raised when a feature is not available

class docplex.util.environment.OverrideEnvironment(new_env=None)

Bases: object

Allows to temporarily replace the default environment.

If the override environment is None, nothing happens and the default environment is not replaced

class docplex.util.environment.SolveDetailsFilter(interval=1)

Bases: object

Default solve detail filter class.

This default class filters details so that there are no more than 1 solve details per second.

filter(details)

Filters the details.

Returns:True if the details are to be published.

docplex.util.environment.add_abort_callback(cb)

Adds the specified callback to the default environment.

The abort callback is called when the script is run on DOcplexcloud and a job abort operation is requested.

Parameters:cb – The abort callback

docplex.util.environment.default_solution_storage_handler(env, solution)

The default solution storage handler.

The storage handler is a function which first argument is the Environment on which a solution should be saved. The solution is a dict containing all the data for an optimization solution.

The storage handler is responsible for storing the solution in the environment.

For each (key, value) pairs of the solution, the default solution storage handler does the following depending of the type of value, in the following order:

• If value is a pandas.DataFrame, then the data frame is saved as an output with the specified name. Note that name must include an extension file for the serialization. See Environment.write_df() for supported formats.
• If value is a bytes, it is saved as binary data with name.
• The value is saved as an output with the name, after it has been converted to JSON.

Parameters:

• env – The Environment
• solution – a dict containing the solution.

docplex.util.environment.get_available_core_count()

Returns the number of cores available for processing if the environment sets a limit, with the default environment.

This number is used in the solving engine as the number of threads.

Returns:The available number of cores or None if the environment does not limit the number of cores.

docplex.util.environment.get_environment()

Returns the Environment object that represents the actual execution environment.

Note: the default environment is the value of the docplex.util.environment.default_environment property.

Returns:An instance of the Environment class that implements methods corresponding to actual execution environment.

docplex.util.environment.get_input_stream(name)

Get an input of the program as a stream (file-like object), with the default environment.

An input of the program is a file that is available in the working directory.

When run on Watson Machine Learning, all input attachments are copied to the working directory before the program is run. get_input_stream lets you open the input attachments of the job.

Parameters:name – Name of the input object. Returns:A file object to read the input from.

docplex.util.environment.get_output_stream(name)

Get a file-like object to write the output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, the output of the program is written as files in the working directory. When run on Watson Machine Learning, the files are attached as output attachments.

The stream is opened in binary mode, and will accept 8 bits data.

Parameters:name – Name of the output object. Returns:A file object to write the output to.

docplex.util.environment.get_parameter(name)

Returns a parameter of the program, with the default environment.

on Watson Machine Learning, this method returns the job parameter whose name is specified.

Parameters:name – The name of the parameter. Returns:The parameter whose name is specified.

docplex.util.environment.make_attachment_name(name)

From name, create an attachment name that is correct for DOcplexcloud.

Attachment filenames in DOcplexcloud has certain restrictions. A file name:

• is limited to 255 characters;
• can include only ASCII characters;
• cannot include the characters /?%*:|”<>, the space character, or the null character; and
• cannot include _ as the first character.

This method replace all unauthorized characters with _, then removing leading ‘_’.

Parameters:name – The original attachment name Returns:An attachment name that conforms to the restrictions. Raises:ValueError if the attachment name is more than 255 characters

docplex.util.environment.maketrans()

Return a translation table usable for str.translate().

If there is only one argument, it must be a dictionary mapping Unicode ordinals (integers) or characters to Unicode ordinals, strings or None. Character keys will be then converted to ordinals. If there are two arguments, they must be strings of equal length, and in the resulting dictionary, each character in x will be mapped to the character at the same position in y. If there is a third argument, it must be a string, whose characters will be mapped to None in the result.

docplex.util.environment.read_df(name, reader=None, **kwargs)

Reads an input of the program as a pandas.DataFrame with the default environment.

pandas must be installed.

name is the name of the input object, as a filename. If a reader is not user provided, the reader used depends on the filename extension.

The default reader used depending on extension are:

• .csv: pandas.read_csv()
• .msg: pandas.read_msgpack()

Parameters:

• name – The name of the input object
• reader – an optional reader function
• **kwargs – additional parameters passed to the reader

Raises:

NotAvailableError – raises this error when pandas is not available.

docplex.util.environment.remove_abort_callback(cb)

Adds the specified callback to the default environment.

The abort callback is called when the script is run on DOcplexcloud and a job abort operation is requested.

Parameters:cb – The abort callback

docplex.util.environment.set_output_attachment(name, filename)

Attach the file which filename is specified as an output of the program.

The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.

When run on premise, filename is copied to the the working directory (if not already there) under the name name.

When run on Watson Machine Learning, the file is attached as output attachment.

Parameters:

• name – Name of the output object.
• filename – The name of the file to attach.

docplex.util.environment.update_solve_details(details)

Update the solve details, with the default environment

You use this method to send solve details to the DOcplexcloud service. If context.solver.auto_publish is set, the underlying solver will automatically update solve details once the solve has finished.

Parameters:details – A dict with solve details as key/value pairs.

docplex.util.environment.write_df(df, name, writer=None, **kwargs)

Write a pandas.DataFrame as an output of the program with the default environment.

pandas must be installed.

name is the name of the input object, as a filename. If a writer is not user provided, the writer used depends on the filename extension.

The default writer used depending on extension are:

• .csv: DataFrame.to_csv()
• .msg: DataFrame.to_msgpack()

Parameters:

• name – The name of the input object
• writer – an optional writer function
• **kwargs – additional parameters passed to the writer

Raises:

NotAvailableError – raises this error when pandas is not available.