#include <basic_manager.h>

Inheritance diagram for tensorflow::serving::BasicManager:

Collaboration diagram for tensorflow::serving::BasicManager:

Classes
struct	Options

Public Types
using	PreLoadHook = std::function< void(const ServableId &)>

using	DoneCallback = std::function< void(const Status &status)>

Public Member Functions
	~BasicManager () override

std::vector< ServableId >	ListAvailableServableIds () const override

Status	GetUntypedServableHandle (const ServableRequest &request, std::unique_ptr< UntypedServableHandle > *untyped_handle) override

std::map< ServableId, std::unique_ptr< UntypedServableHandle > >	GetAvailableUntypedServableHandles () const override

Status	ManageServable (ServableData< std::unique_ptr< Loader >> servable)

template<typename T >
Status	ManageServableWithAdditionalState (ServableData< std::unique_ptr< Loader >> servable, std::unique_ptr< T > additional_state)

Status	StopManagingServable (const ServableId &id)

std::vector< string >	GetManagedServableNames () const

template<typename T = std::nullptr_t>
std::vector< ServableStateSnapshot< T > >	GetManagedServableStateSnapshots (const string &servable_name) const

template<typename T = std::nullptr_t>
absl::optional< ServableStateSnapshot< T > >	GetManagedServableStateSnapshot (const ServableId &id)

template<typename T >
T *	GetAdditionalServableState (const ServableId &id)

void	LoadServable (const ServableId &id, DoneCallback done_callback)

void	CancelLoadServableRetry (const ServableId &id)

void	UnloadServable (const ServableId &id, DoneCallback done_callback)

Public Member Functions inherited from tensorflow::serving::Manager
template<typename T >
std::map< ServableId, ServableHandle< T > >	GetAvailableServableHandles () const

template<typename T >
Status	GetServableHandle (const ServableRequest &request, ServableHandle< T > *const handle)

Static Public Member Functions
static Status	Create (Options options, std::unique_ptr< BasicManager > *manager)

Friends
class	AspiredVersionsManager

class	test_util::BasicManagerTestAccess

Detailed Description

Helps manage the lifecycle of servables including loading, serving and unloading them. The manager accepts servables in the form of Loaders.

We start managing a servable through one of the ManageServable* methods. You can go on to load the servable after this by calling LoadServable(). Loading will also make the servable available to serve. Once you decide to unload it, you can call UnloadServable() on it, which will make it unavailable to serve, then unload the servable.

Servables are retained until StopManagingServable() is called. This allows a higher level manager with more information to decide when it's safe to forget about a servable.

BasicManager tracks the resources (e.g. RAM) used by loaded servables, and only allows loading new servables that fit within the overall resource pool.

BasicManager can be configured to use a thread-pool to do it's load and unloads. This makes the LoadServable() and UnloadServable() methods schedule the load/unloads rather than executing them synchronously. If there are more pending load/unloads than threads in the thread pool, they are processed in FIFO order.

In the presence of loaders that over-estimate their servables' resource needs and/or only bind their servables' resources to device instances, load/unload concurrency can be reduced below the thread-pool size. That is because we may have to wait for one servable's load/unload to finish to pin down the resource availability for loading another servable.

REQUIRES:

Order of method calls - ManageServable() (and variants) -> LoadServable() -> UnloadServable() -> StopManagingServable().
Do not schedule concurrent load and unloads of the same servable.
Do not call load or unload multiple times on the same servable.

This class is thread-safe.

Example usage:

const ServableId id = {kServableName, 0};
std::unique_ptr&lt;Loader> loader = ...;
...
BasicManager manager;
TF_CHECK_OK(manager.ManageServable(
  CreateServableData(id, std::move(loader))));
TF_CHECK_OK(manager.LoadServable(id));

...
TF_CHECK_OK(manager.GetServableHandle(
    ServableRequest::Latest(kServableName), &handle));
...

TF_CHECK_OK(manager.UnloadServable(id));
TF_CHECK_OK(manager.StopManagingServable(id));

Definition at line 106 of file basic_manager.h.

Member Typedef Documentation

◆ DoneCallback

using tensorflow::serving::BasicManager::DoneCallback = std::function<void(const Status& status)>

Callback called at the end of {Load,Unload}Servable(). We pass in the status of the operation to the callback.

Definition at line 232 of file basic_manager.h.

Constructor & Destructor Documentation

◆ ~BasicManager()

tensorflow::serving::BasicManager::~BasicManager ( )

override

If configured to use a load/unload thread-pool, waits until all scheduled loads and unloads have finished and then destroys the set of threads.

Definition at line 270 of file basic_manager.cc.

Member Function Documentation

◆ CancelLoadServableRetry()

void tensorflow::serving::BasicManager::CancelLoadServableRetry ( const ServableId & id )

Cancels retrying the servable load during LoadServable() by replacing the LoaderHarness::should_retry with a function that always returns false. Does nothing if the servable isn't managed.

If the retries are cancelled, the servable goes into a state dependent on the last Load() called on it. If the last Load() was successful, it will be in state kReady, else in kError.

Definition at line 532 of file basic_manager.cc.

◆ GetAdditionalServableState()

template<typename T >

T * tensorflow::serving::BasicManager::GetAdditionalServableState ( const ServableId & id )

Returns: the additional state for the servable. Returns nullptr if there is no additional state setup or if there is a type mismatch between what was setup and what is being asked for.

REQUIRES: This manager should have been managing this servable already, else we return nullptr.

Definition at line 578 of file basic_manager.h.

◆ GetManagedServableNames()

std::vector< string > tensorflow::serving::BasicManager::GetManagedServableNames ( ) const

Returns: the names of all the servables managed by this manager. The names will be duplicate-free and not in any particular order.

Definition at line 477 of file basic_manager.cc.

◆ GetManagedServableStateSnapshot()

template<typename T >

absl::optional< ServableStateSnapshot< T > > tensorflow::serving::BasicManager::GetManagedServableStateSnapshot ( const ServableId & id )

Returns: the state snapshot of a particular servable-id managed by this manager if available.

REQUIRES: This manager should have been managing this servable already, else we return nullopt.

Definition at line 567 of file basic_manager.h.

◆ GetManagedServableStateSnapshots()

template<typename T >

std::vector< ServableStateSnapshot< T > > tensorflow::serving::BasicManager::GetManagedServableStateSnapshots ( const string & servable_name ) const

Returns: the state snapshots of all the servables of a particular stream, managed by this manager.

T is the additional-state type, if any.

Definition at line 551 of file basic_manager.h.

◆ ListAvailableServableIds()

std::vector< ServableId > tensorflow::serving::BasicManager::ListAvailableServableIds ( ) const

overridevirtual

Gets a list of all available servable ids, i.e. each of these can be retrieved using GetServableHandle.

Implements tensorflow::serving::Manager.

Definition at line 311 of file basic_manager.cc.

◆ LoadServable()

void tensorflow::serving::BasicManager::LoadServable	(	const ServableId &	id,
		DoneCallback	done_callback
	)

Loads the servable with this id, and updates the serving map too. Calls done_callback with ok iff the servable was loaded successfully, else returns an error status.

If using a thread-pool, this method transitions the servable harness to kLoading state, schedules the load and returns, otherwise it completes the load before returning.

REQUIRES: This manager should have been managing this servable already, for it to be loaded, else we call done_callback with an error status. Do not call this multiple times on the same servable. Only one of those will succeed and the rest will fail with an error status.

Definition at line 523 of file basic_manager.cc.

◆ ManageServable()

Status tensorflow::serving::BasicManager::ManageServable ( ServableData< std::unique_ptr< Loader >> servable )

Starts managing the servable.

Returns an error if given a servable that is already being managed.

If servable is in an error state, this method does not return an error. Instead, the manager accepts the servable, puts it in state kError (with a notification sent to the event bus), and then immediately stops managing it. This behavior facilitates uniform handling of errors that occur in sources (e.g. invalid file path to servable data) and ones that occur in the manager (e.g. insufficient resources to load servable).

Definition at line 380 of file basic_manager.cc.

◆ ManageServableWithAdditionalState()

template<typename T >

Status tensorflow::serving::BasicManager::ManageServableWithAdditionalState	(	ServableData< std::unique_ptr< Loader >>	servable,
		std::unique_ptr< T >	additional_state
	)

Similar to the above method, but callers, usually other managers built on top of this one, can associate additional state with the servable. Additional state may be ACL or lifetime metadata for that servable. The ownership of the state is transferred to this class.

Definition at line 536 of file basic_manager.h.

◆ StopManagingServable()

Status tensorflow::serving::BasicManager::StopManagingServable ( const ServableId & id )

Tells the manager to stop managing this servable. Requires that the servable is currently being managed and that its state is one of {kNew, kError, kDisabled}.

Definition at line 390 of file basic_manager.cc.

◆ UnloadServable()

void tensorflow::serving::BasicManager::UnloadServable	(	const ServableId &	id,
		DoneCallback	done_callback
	)

Unloads the servable with this id, and updates the serving map too. Calls done_callback with ok iff the servable was unloaded successfully, else returns an error status.

If using a thread-pool, this method transitions the servable harness to kQuiescing state, schedules the unload and returns, otherwise it completes the unload before returning.

REQUIRES: This manager should have loaded and made this servable available, for it to be unloaded, else calls done_callback with an error status. Do not call this multiple times on the same servable. Only one of those will succeed and the rest will fail with an error status.

Definition at line 563 of file basic_manager.cc.

The documentation for this class was generated from the following files:

tensorflow_serving/core/basic_manager.h
tensorflow_serving/core/basic_manager.cc

Classes

Public Types

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Member Typedef Documentation

◆ DoneCallback

Constructor & Destructor Documentation

◆ ~BasicManager()

Member Function Documentation

◆ CancelLoadServableRetry()

◆ GetAdditionalServableState()

◆ GetManagedServableNames()

◆ GetManagedServableStateSnapshot()

◆ GetManagedServableStateSnapshots()

◆ ListAvailableServableIds()

◆ LoadServable()

◆ ManageServable()

◆ ManageServableWithAdditionalState()

◆ StopManagingServable()

◆ UnloadServable()