docplex.util.environment module¶
Representation of the DOcplex solving environment.
This module handles the various elements that allow an optimization program to run independently from the solving environment. This environment may be:
- on premise, using a local version of CPLEX Optimization Studio to solve MP problems, or
- on DOcplexcloud, with the Python program running inside the Python Worker.
- on Decision Optimization in Watson Machine Learning.
As much as possible, the adaptation to the solving environment is automatic. The functions that are presented here are useful for handling very specific use cases.
The following code is a program that sums its input (sum.py
):
import json
import docplex.util.environment as environment
sum = 0
# open program input named "data.txt" and sum the contents
with environment.get_input_stream("data.txt") as input:
for i in input.read().split():
sum += int(i)
# write the result as a simple json in program output "solution.json"
with environment.get_output_stream("solution.json") as output:
output.write(json.dumps({'result': sum}))
Let’s put some data in a data.txt
file:
4 7 8
19
When you run sum.py
with a Python interpreter, it opens the data.txt
file and sums all of the integers
in that file. The result is saved as a JSON fragment in file solution.json
:
$ python sum.py
$ more solution.json
{"result": 38}
Environment representation can be accessed with different ways:
direct object method calls, after retrieving an instance using
docplex.util.environment.get_environment()
and using methods ofdocplex.util.environment.Environment
.
- using the function in package docplex.util.environment. They will call
the corresponding methods of Environment in the platform default_environment:
docplex.util.environment.get_input_stream()
docplex.util.environment.get_output_stream()
docplex.util.environment.read_df()
docplex.util.environment.write_df()
docplex.util.environment.get_available_core_count()
docplex.util.environment.get_parameter()
docplex.util.environment.update_solve_details()
docplex.util.environment.add_abort_callback()
docplex.util.environment.remove_abort_callback()
-
class
docplex.util.environment.
AbstractLocalEnvironment
[source]¶ Bases:
docplex.util.environment.Environment
-
get_available_core_count
()[source]¶ Returns the number of cores available for processing if the environment sets a limit.
This number is used in the solving engine as the number of threads.
Returns: The available number of cores or None
if the environment does not limit the number of cores.
-
get_input_stream
(name)[source]¶ Get an input of the program as a stream (file-like object).
An input of the program is a file that is available in the working directory.
When run on DOcplexcloud, all input attachments are copied to the working directory before the program is run.
get_input_stream
lets you open the input attachments of the job.Parameters: name – Name of the input object. Returns: A file object to read the input from.
-
get_output_stream
(name)[source]¶ Get a file-like object to write the output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise, the output of the program is written as files in the working directory. When run on DOcplexcloud, the files are attached as output attachments.
The stream is opened in binary mode, and will accept 8 bits data.
Parameters: name – Name of the output object. Returns: A file object to write the output to.
-
get_parameter
(name)[source]¶ Returns a parameter of the program.
On DOcplexcloud, this method returns the job parameter whose name is specified. On local solver, this method returns the environment variable whose name is specified.
Parameters: name – The name of the parameter. Returns: The parameter whose name is specified or None if the parameter does not exists.
-
get_parameters
()[source]¶ Returns a dict containing all parameters of the program.
On DOcplexcloud, this method returns the job parameters. On local solver, this method returns
os.environ
.Returns: The job parameters
-
set_output_attachment
(name, filename)[source]¶ Attach the file which filename is specified as an output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise,
filename
is copied to the the working directory (if not already there) under the namename
.When run on DOcplexcloud, the file is attached as output attachment.
Parameters: - name – Name of the output object.
- filename – The name of the file to attach.
-
-
class
docplex.util.environment.
Environment
[source]¶ Bases:
object
Methods for interacting with the execution environment.
Internally, the
docplex
package provides the appropriate implementation according to the actual execution environment. The correct instance of this class is returned by the methoddocplex.util.environment.get_environment()
that is provided in this module.-
abort_callbacks
¶ A list of callbacks that are called when the script is run on DOcplexcloud and a job abort operation is requested. You add your own callback using:
env.abort_callbacks += [your_cb]
or:
env.abort_callbacks.append(your_cb)
You remove a callback using:
env.abort_callbacks.remove(your_cb)
-
solution_storage_handler
¶ A function called when a solution is to be stored. The storage handler is a function which first argument is the
Environment
on which a solution should be saved. The solution is a dict containing all the data for an optimization solution. The default isdefault_solution_storage_handler()
.
-
record_history_fields
¶ Fields which history is to be kept
-
record_history_size
¶ maximum number of records in history
-
record_interval
¶ min time between to history records
-
get_available_core_count
()[source]¶ Returns the number of cores available for processing if the environment sets a limit.
This number is used in the solving engine as the number of threads.
Returns: The available number of cores or None
if the environment does not limit the number of cores.
-
get_engine_log_level
()[source]¶ Returns the engine log level as set by job parameter oaas.engineLogLevel.
oaas.engineLogLevel values are: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
The mapping to logging levels in python are:
- OFF: None
- SEVERE: logging.ERROR
- WARNING: logging.WARNING
- INFO, CONFIG: logging.INFO
- FINE, FINER, FINEST: logging.DEBUG
- ALL: logging.DEBUG
All other values are considered invalid values and will return None.
Returns: The logging level or None if not set (off)
-
get_input_stream
(name)[source]¶ Get an input of the program as a stream (file-like object).
An input of the program is a file that is available in the working directory.
When run on DOcplexcloud, all input attachments are copied to the working directory before the program is run.
get_input_stream
lets you open the input attachments of the job.Parameters: name – Name of the input object. Returns: A file object to read the input from.
-
get_output_stream
(name)[source]¶ Get a file-like object to write the output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise, the output of the program is written as files in the working directory. When run on DOcplexcloud, the files are attached as output attachments.
The stream is opened in binary mode, and will accept 8 bits data.
Parameters: name – Name of the output object. Returns: A file object to write the output to.
-
get_parameter
(name)[source]¶ Returns a parameter of the program.
On DOcplexcloud, this method returns the job parameter whose name is specified. On local solver, this method returns the environment variable whose name is specified.
Parameters: name – The name of the parameter. Returns: The parameter whose name is specified or None if the parameter does not exists.
-
get_parameters
()[source]¶ Returns a dict containing all parameters of the program.
On DOcplexcloud, this method returns the job parameters. On local solver, this method returns
os.environ
.Returns: The job parameters
-
get_stop_callback
()[source]¶ Returns the stop callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.
You can also use the
stop_callback
property to get the callback.Deprecated since 2.4 - Use the abort_callbacks property instead’)
-
is_debug_mode
()[source]¶ Returns true if the engine should run in debug mode.
This is equivalent to
env.get_engine_log_level() <= logging.DEBUG
-
publish_solve_details
(details)[source]¶ Actually publish the solve specified details.
Returns: The published details
-
read_df
(name, reader=None, **kwargs)[source]¶ Reads an input of the program as a
pandas.DataFrame
.pandas
must be installed.name
is the name of the input object, as a filename. If a reader is not user provided, the reader used depends on the filename extension.The default reader used depending on extension are:
.csv
:pandas.read_csv()
.msg
:pandas.read_msgpack()
Parameters: - name – The name of the input object
- reader – an optional reader function
- **kwargs – additional parameters passed to the reader
Raises: NotAvailableError
– raises this error whenpandas
is not available.
-
set_output_attachment
(name, filename)[source]¶ Attach the file which filename is specified as an output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise,
filename
is copied to the the working directory (if not already there) under the namename
.When run on DOcplexcloud, the file is attached as output attachment.
Parameters: - name – Name of the output object.
- filename – The name of the file to attach.
-
set_stop_callback
(cb)[source]¶ Sets a callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.
You can also use the
stop_callback
property to set the callback.Deprecated since 2.4 - Use self.abort_callbacks += [cb] instead’
Parameters: cb – The callback function
-
stop_callback
¶ Returns the stop callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.
You can also use the
stop_callback
property to get the callback.Deprecated since 2.4 - Use the abort_callbacks property instead’)
-
store_solution
(solution)[source]¶ Stores the specified solution.
This method guarantees that the solution is fully saved if the model is running on DOcplexcloud python worker and an abort of the job is triggered.
For each (key, value) pairs of the solution, the default solution storage handler does the following depending of the type of value, in the following order:
- If value is a pandas.DataFrame, then the data frame is saved
as an output with the specified name. Note that name must include
an extension file for the serialization. See
Environment.write_df()
for supported formats. - If value is a bytes, it is saved as binary data with name.
- The value is saved as an output with the name, after it has been converted to JSON.
Parameters: solution – a dict containing the solution. - If value is a pandas.DataFrame, then the data frame is saved
as an output with the specified name. Note that name must include
an extension file for the serialization. See
-
update_solve_details
(details)[source]¶ Update the solve details.
You use this method to send solve details to the solve service. If
context.solver.auto_publish
is set, the underlying solver will automatically update solve details once the solve has finished.This method might filter details and publish them with rate limitations. It actually publish de details by calling publish_solve_details.
Parameters: details – A dict
with solve details as key/value pairs.
-
write_df
(df, name, writer=None, **kwargs)[source]¶ Write a
pandas.DataFrame
as an output of the program.pandas
must be installed.name
is the name of the input object, as a filename. If a writer is not user provided, the writer used depends on the filename extension.This currently only supports csv output.
Parameters: - name – The name of the input object
- writer – an optional writer function
- **kwargs – additional parameters passed to the writer
Raises: NotAvailableError
– raises this error whenpandas
is not available.
-
-
exception
docplex.util.environment.
NotAvailableError
[source]¶ Bases:
Exception
The exception raised when a feature is not available
-
class
docplex.util.environment.
OverrideEnvironment
(new_env=None)[source]¶ Bases:
object
Allows to temporarily replace the default environment.
If the override environment is None, nothing happens and the default environment is not replaced
-
class
docplex.util.environment.
SolveDetailsFilter
(interval=1)[source]¶ Bases:
object
Default solve detail filter class.
This default class filters details so that there are no more than 1 solve details per second.
-
class
docplex.util.environment.
WorkerEnvironment
(solve_hook)[source]¶ Bases:
docplex.util.environment.Environment
-
get_available_core_count
()[source]¶ Returns the number of cores available for processing if the environment sets a limit.
This number is used in the solving engine as the number of threads.
Returns: The available number of cores or None
if the environment does not limit the number of cores.
-
get_input_stream
(name)[source]¶ Get an input of the program as a stream (file-like object).
An input of the program is a file that is available in the working directory.
When run on DOcplexcloud, all input attachments are copied to the working directory before the program is run.
get_input_stream
lets you open the input attachments of the job.Parameters: name – Name of the input object. Returns: A file object to read the input from.
-
get_output_stream
(name)[source]¶ Get a file-like object to write the output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise, the output of the program is written as files in the working directory. When run on DOcplexcloud, the files are attached as output attachments.
The stream is opened in binary mode, and will accept 8 bits data.
Parameters: name – Name of the output object. Returns: A file object to write the output to.
-
get_parameter
(name)[source]¶ Returns a parameter of the program.
On DOcplexcloud, this method returns the job parameter whose name is specified. On local solver, this method returns the environment variable whose name is specified.
Parameters: name – The name of the parameter. Returns: The parameter whose name is specified or None if the parameter does not exists.
-
get_parameters
()[source]¶ Returns a dict containing all parameters of the program.
On DOcplexcloud, this method returns the job parameters. On local solver, this method returns
os.environ
.Returns: The job parameters
-
get_stop_callback
()[source]¶ Returns the stop callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.
You can also use the
stop_callback
property to get the callback.Deprecated since 2.4 - Use the abort_callbacks property instead’)
-
publish_solve_details
(details)[source]¶ Actually publish the solve specified details.
Returns: The published details
-
set_output_attachment
(name, filename)[source]¶ Attach the file which filename is specified as an output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise,
filename
is copied to the the working directory (if not already there) under the namename
.When run on DOcplexcloud, the file is attached as output attachment.
Parameters: - name – Name of the output object.
- filename – The name of the file to attach.
-
set_stop_callback
(cb)[source]¶ Sets a callback that is called when the script is run on DOcplexcloud and a job abort operation is requested.
You can also use the
stop_callback
property to set the callback.Deprecated since 2.4 - Use self.abort_callbacks += [cb] instead’
Parameters: cb – The callback function
-
-
docplex.util.environment.
add_abort_callback
(cb)[source]¶ Adds the specified callback to the default environment.
The abort callback is called when the script is run on DOcplexcloud and a job abort operation is requested.
Parameters: cb – The abort callback
-
docplex.util.environment.
default_solution_storage_handler
(env, solution)[source]¶ The default solution storage handler.
The storage handler is a function which first argument is the
Environment
on which a solution should be saved. The solution is a dict containing all the data for an optimization solution.The storage handler is responsible for storing the solution in the environment.
For each (key, value) pairs of the solution, the default solution storage handler does the following depending of the type of value, in the following order:
- If value is a pandas.DataFrame, then the data frame is saved
as an output with the specified name. Note that name must include
an extension file for the serialization. See
Environment.write_df()
for supported formats. - If value is a bytes, it is saved as binary data with name.
- The value is saved as an output with the name, after it has been converted to JSON.
Parameters: - env – The
Environment
- solution – a dict containing the solution.
- If value is a pandas.DataFrame, then the data frame is saved
as an output with the specified name. Note that name must include
an extension file for the serialization. See
-
docplex.util.environment.
get_available_core_count
()[source]¶ Returns the number of cores available for processing if the environment sets a limit, with the default environment.
This number is used in the solving engine as the number of threads.
Returns: The available number of cores or None
if the environment does not limit the number of cores.
-
docplex.util.environment.
get_environment
()[source]¶ Returns the Environment object that represents the actual execution environment.
Note: the default environment is the value of the
docplex.util.environment.default_environment
property.Returns: An instance of the Environment
class that implements methods corresponding to actual execution environment.
-
docplex.util.environment.
get_input_stream
(name)[source]¶ Get an input of the program as a stream (file-like object), with the default environment.
An input of the program is a file that is available in the working directory.
When run on DOcplexcloud, all input attachments are copied to the working directory before the program is run.
get_input_stream
lets you open the input attachments of the job.Parameters: name – Name of the input object. Returns: A file object to read the input from.
-
docplex.util.environment.
get_output_stream
(name)[source]¶ Get a file-like object to write the output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise, the output of the program is written as files in the working directory. When run on DOcplexcloud, the files are attached as output attachments.
The stream is opened in binary mode, and will accept 8 bits data.
Parameters: name – Name of the output object. Returns: A file object to write the output to.
-
docplex.util.environment.
get_parameter
(name)[source]¶ Returns a parameter of the program, with the default environment.
On DOcplexcloud, this method returns the job parameter whose name is specified.
Parameters: name – The name of the parameter. Returns: The parameter whose name is specified.
-
docplex.util.environment.
make_attachment_name
(name)[source]¶ From name, create an attachment name that is correct for DOcplexcloud.
Attachment filenames in DOcplexcloud has certain restrictions. A file name:
- is limited to 255 characters;
- can include only ASCII characters;
- cannot include the characters /?%*:|”<>, the space character, or the null character; and
- cannot include _ as the first character.
This method replace all unauthorized characters with _, then removing leading ‘_’.
Parameters: name – The original attachment name Returns: An attachment name that conforms to the restrictions. Raises: ValueError if the attachment name is more than 255 characters
-
docplex.util.environment.
maketrans
()¶ Return a translation table usable for str.translate().
If there is only one argument, it must be a dictionary mapping Unicode ordinals (integers) or characters to Unicode ordinals, strings or None. Character keys will be then converted to ordinals. If there are two arguments, they must be strings of equal length, and in the resulting dictionary, each character in x will be mapped to the character at the same position in y. If there is a third argument, it must be a string, whose characters will be mapped to None in the result.
-
docplex.util.environment.
read_df
(name, reader=None, **kwargs)[source]¶ Reads an input of the program as a
pandas.DataFrame
with the default environment.pandas
must be installed.name
is the name of the input object, as a filename. If a reader is not user provided, the reader used depends on the filename extension.The default reader used depending on extension are:
.csv
:pandas.read_csv()
.msg
:pandas.read_msgpack()
Parameters: - name – The name of the input object
- reader – an optional reader function
- **kwargs – additional parameters passed to the reader
Raises: NotAvailableError
– raises this error whenpandas
is not available.
-
docplex.util.environment.
remove_abort_callback
(cb)[source]¶ Adds the specified callback to the default environment.
The abort callback is called when the script is run on DOcplexcloud and a job abort operation is requested.
Parameters: cb – The abort callback
-
docplex.util.environment.
set_output_attachment
(name, filename)[source]¶ Attach the file which filename is specified as an output of the program.
The file is recorded as being part of the program output. This method can be called multiple times if the program contains multiple output objects.
When run on premise,
filename
is copied to the the working directory (if not already there) under the namename
.When run on DOcplexcloud, the file is attached as output attachment.
Parameters: - name – Name of the output object.
- filename – The name of the file to attach.
-
docplex.util.environment.
update_solve_details
(details)[source]¶ Update the solve details, with the default environment
You use this method to send solve details to the DOcplexcloud service. If
context.solver.auto_publish
is set, the underlying solver will automatically update solve details once the solve has finished.Parameters: details – A dict
with solve details as key/value pairs.
-
docplex.util.environment.
write_df
(df, name, writer=None, **kwargs)[source]¶ Write a
pandas.DataFrame
as an output of the program with the default environment.pandas
must be installed.name
is the name of the input object, as a filename. If a writer is not user provided, the writer used depends on the filename extension.The default writer used depending on extension are:
.csv
:DataFrame.to_csv()
.msg
:DataFrame.to_msgpack()
Parameters: - name – The name of the input object
- writer – an optional writer function
- **kwargs – additional parameters passed to the writer
Raises: NotAvailableError
– raises this error whenpandas
is not available.