AWS Sagemaker model

[aatish-shinde]

Hi

I have a Mistral model deployed on AWS Sagemaker. Is there a way I can use the Sagemaker LLM with MCP client?

I have this class below. I would like to use it with MCP. Is it possible?

def get_streaming_response(self, prompt):
    """
    Sends a structured prompt to the SageMaker LLM endpoint and streams the response.

    :param prompt: The structured prompt formatted as a JSON list.
    :return: The streamed response as a string.
    """
    try:
        # Define inference parameters with streaming enabled
        inference_params = {
            "do_sample": True,
            "temperature": 0.1,
            "top_k": 50,
            "max_new_tokens": 512,
            "repetition_penalty": 1.03,
            "stop": ["</s>", "<|system|>", "<|user|>", "<|assistant|>"],
            "return_full_text": False
        }

        body = json.dumps({"inputs": prompt, "parameters": inference_params, "stream": True})

        # Invoke SageMaker endpoint with response streaming
        response = self.client.invoke_endpoint_with_response_stream(
            EndpointName=self.endpoint_name,
            Body=body,
            ContentType="application/json"
        )

        event_stream = response["Body"]

        return event_stream

    except Exception as e:
        self.logger.error(f"🚨 SageMaker error: {e}")
        return "Error processing request. Please try again."

Thanks
Atish

[SRockt1]

Hallo

Ich habe ein Mistral-Modell auf AWS Sagemaker bereitgestellt. Gibt es eine Möglichkeit, das Sagemaker LLM mit dem MCP-Client zu verwenden?

Ich habe die folgende Klasse. Ich möchte sie mit MCP verwenden. Ist das möglich?

def get_streaming_response(self, prompt):
    """
    Sends a structured prompt to the SageMaker LLM endpoint and streams the response.

    :param prompt: The structured prompt formatted as a JSON list.
    :return: The streamed response as a string.
    """
    try:
        # Define inference parameters with streaming enabled
        inference_params = {
            "do_sample": True,
            "temperature": 0.1,
            "top_k": 50,
            "max_new_tokens": 512,
            "repetition_penalty": 1.03,
            "stop": ["</s>", "<|system|>", "<|user|>", "<|assistant|>"],
            "return_full_text": False
        }

        body = json.dumps({"inputs": prompt, "parameters": inference_params, "stream": True})

        # Invoke SageMaker endpoint with response streaming
        response = self.client.invoke_endpoint_with_response_stream(
            EndpointName=self.endpoint_name,
            Body=body,
            ContentType="application/json"
        )

        event_stream = response["Body"]

        return event_stream

    except Exception as e:
        self.logger.error(f"🚨 SageMaker error: {e}")
        return "Error processing request. Please try again."

Danke , Atish

ok