#include <server_core.h>
Classes | |
| struct | Options |
| Options for configuring a ServerCore object. More... | |
Public Types | |
| using | PreLoadHook = AspiredVersionsManager::PreLoadHook |
| using | ServableStateMonitorCreator = std::function< Status(EventBus< ServableState > *event_bus, std::unique_ptr< ServableStateMonitor > *monitor)> |
| using | CustomModelConfigLoader = std::function< Status(const ::google::protobuf::Any &any, EventBus< ServableState > *event_bus, UniquePtrWithDeps< AspiredVersionsManager > *manager)> |
| using | ServerRequestLoggerUpdater = std::function< Status(const ModelServerConfig &, ServerRequestLogger *)> |
| Function signature used to update the server_request_logger. | |
Public Member Functions | |
| std::vector< ServableId > | ListAvailableServableIds () const override |
| virtual Status | ReloadConfig (const ModelServerConfig &config) TF_LOCKS_EXCLUDED(config_mu_) |
| virtual ServableStateMonitor * | servable_state_monitor () const |
| Returns ServableStateMonitor that can be used to query servable states. | |
| template<typename T > | |
| Status | GetServableHandle (const ModelSpec &model_spec, ServableHandle< T > *const handle) |
| virtual Status | GetServableHandle (const ModelSpec &model_spec, ServableHandle< Servable > *const handle) |
| template<typename T > | |
| std::map< ServableId, ServableHandle< T > > | GetAvailableServableHandles () const |
| virtual Status | Log (const google::protobuf::Message &request, const google::protobuf::Message &response, const LogMetadata &log_metadata) |
| template<typename Request , typename Response > | |
| std::unique_ptr< StreamLogger< Request, Response > > | StartLoggingStream (const LogMetadata &log_metadata, ServerRequestLogger::CreateStreamLoggerFn< Request, Response > create_stream_logger_fn) |
| internal::PredictResponseTensorSerializationOption | predict_response_tensor_serialization_option () const |
| bool | enable_cors_support () const |
 Public Member Functions inherited from tensorflow::serving::Manager | |
| template<typename T > | |
| std::map< ServableId, ServableHandle< T > > | GetAvailableServableHandles () const |
| template<typename T > | |
| Status | GetServableHandle (const ServableRequest &request, ServableHandle< T > *const handle) |
Static Public Member Functions | |
| static Status | Create (Options options, std::unique_ptr< ServerCore > *core) |
Protected Member Functions | |
| ServerCore (Options options) | |
Friends | |
| class | test_util::ServerCoreTestAccess |
Detailed Description
ServerCore contains state and helper methods enabling the building of ModelServers that support multiple interfaces. All functionality in ServerCore is independent of any domain specific APIs and independent of platforms.
In terms of state, ServerCore is initialized with and retains a static ModelServerConfig, from which it bootstraps an AspiredVersionsManager and auxiliary data structures to support efficient serving.
Interfaces built above ServerCore, e.g. RPC service implementations, will remain stateless and will perform all lookups of servables (models) via ServerCore.
Definition at line 74 of file server_core.h.
Member Typedef Documentation
◆ CustomModelConfigLoader
| using tensorflow::serving::ServerCore::CustomModelConfigLoader = std::function<Status( const ::google::protobuf::Any& any, EventBus<ServableState>* event_bus, UniquePtrWithDeps<AspiredVersionsManager>* manager)> |
A function that's responsible for instantiating and connecting the necessary custom sources and source adapters to the manager based on a passed in config (any). The expected pattern is that ownership of the created sources/source adapters can be transferred to the manager.
Definition at line 87 of file server_core.h.
Member Function Documentation
◆ Create()
|
static |
Creates a ServerCore instance with all the models and sources per the ModelServerConfig.
For models statically configured with ModelConfigList, waits for them to be made available (or hit an error) for serving before returning. Returns an error status if any such model fails to load.
Definition at line 231 of file server_core.cc.
◆ GetServableHandle()
|
inline |
Returns a ServableHandle given a ModelSpec. Returns error if no such Servable is available – e.g. not yet loaded, has been quiesced/unloaded, etc. Callers may assume that an OK status indicates a non-null handle.
IMPORTANT: The caller should only hold on to a handle for a short time, for example for the duration of a single request. Holding a handle for a long period of time will prevent servable loading and unloading.
If 'options_.allow_version_labels==true', recognizes two specific model version labels – "stable" and "canary" – and resolves them to the smallest and largest available version, respectively.
Definition at line 267 of file server_core.h.
◆ ListAvailableServableIds()
|
inlineoverridevirtual |
Gets a list of all available servable ids, i.e. each of these can be retrieved using GetServableHandle.
Implements tensorflow::serving::Manager.
Definition at line 234 of file server_core.h.
◆ Log()
|
inlinevirtual |
Writes the log for the particular request, response and metadata, if we decide to sample it and if request-logging was configured for the particular model.
Definition at line 299 of file server_core.h.
◆ ReloadConfig()
|
virtual |
Updates the server core with all the models and sources per the ModelServerConfig. Like Create(), waits for all statically configured servables to be made available before returning, and returns an error if any such model fails to load. (Does not necessarily wait for models removed from the config to finish unloading; that may occur asynchronously.)
IMPORTANT: It is only legal to call this method more than once if using ModelConfigList (versus custom model config).
Definition at line 447 of file server_core.cc.
The documentation for this class was generated from the following files:
- tensorflow_serving/model_servers/server_core.h
- tensorflow_serving/model_servers/server_core.cc
