Tutorial 5: Endpoints (Query a Large Language Model)
Tutorial 5: Endpoints (Query a Large Language Model)¶
In this tutorial, we will build a simple demo interface to ask a large language model (LLM) questions.
Through this tutorial, you will learn about:
the concept of endpoints
how to pass endpoints to components to run on events
manifest, a server-client manager for running LLMs
You will need to install
manifest and set up your OPENAI API key:
# Install **manifest** pip install manifest-ml # Set up your OpenAI key. export OPENAI_API_KEY="your-api-key"
To get started, run the tutorial demo script
mk demo tutorial-qa
You should see the tutorial app when you open the link in your browser. Let’s break down the code in the demo script.
In this interface, we want to ask the LLM for an answer to our question when we press a button.
This requires having
An LLM to return answers to our questions
An input text field to ask our question
A Python API that runs the LLM
A button to call the Python API on click
(Aside) Setting up LLMs with Manifest¶
Manifest is a Python library that provides a convenient interface to call foundation model APIs, e.g. large language models from OpenAI, Cohere, Together, etc.
Let’s set up manifest to connect to OpenAI.
manifest = Manifest( client_name="openai", client_connection=os.getenv("OPENAI_API_KEY"), engine="text-davinci-003", )
Running a prompt through manifest will be easy:
manifest.run("What is the meaning of life?", max_tokens=100)
That’s it! You can refer to the Manifest codebase to learn more about how to use it. Let’s now move on to the rest of the application.
Adding an Input TextField Component¶
Let’s add a text field where a user can type out their questions.
Meerkat provides a
Textbox component that we can use.
textbox = mk.gui.Textbox()
This component will do a few things:
create a text field on the frontend that the user can type into
synchronize the value of this text field with a
Storeobject on the Python backend
textbox.text will be a
Store, a Meerkat data object that is synchronized with the value of the text field on the frontend. This will be important when we want to pass the value of the text field to our LLM API.
type(textbox.text): <class 'meerkat.interactive.graph.store.Store'>
Now that we have our LLM and our input field, we want to write a Python function that will run the LLM on the input.
This function has to be special in a few ways:
It should be able to take in the value of the input field
It should run when an event on the frontend happens, e.g. a button is clicked
To do this, we can use the
@mk.endpoint decorator. This decorator will convert a Python function into an
Endpoint object, which is a special type of Python function that can be passed to components to run on events.
@mk.endpoint() def get_answer(question: str, answer: mk.Store): response = manifest.run(question, max_tokens=100) answer.set(response) return answer
If you print this endpoint out, you’ll see that it the decorator has converted the
get_answer function into an
get_answer: Endpoint(id=__mkid__dee421331bd545fba5135210c020a607, name=get_answer, prefix=None, route=None)
Notice that in addition to the
question, we also pass in an
answer argument. This allows us to update the
answer store with the response from the LLM. When the store is updated, it will tell all frontend components that depend on this store to also update their values.
Running Endpoints with Component Events¶
Now that we have our endpoint, we want to run it when a button is clicked.
We can do this by passing the endpoint to the
on_click argument of a
# The store to keep track of the LLM answer. answer = mk.Store("") button = mk.gui.Button( title="Ask", on_click=get_answer.partial(question=textbox.text, answer=answer), )
When the button is clicked, the
get_answer endpoint will be called with the value of the
textbox.text store as the
We can simulate this button click by running the
on_click endpoint directly. Since it’s an
Endpoint object, we can call it with the
Different components expose different events that can be tied to endpoints. All endpoints start with
on_ and are followed by the name of the event. For example, the
on_click event is tied to the
click event of a
Putting it all together¶
Now we display these different components on the frontend.
page = mk.gui.Page( component=mk.gui.html.flexcol([textbox, button, mk.gui.Text(answer)]), id="tutorial-query-llm", )
When you run the app, you should see the following interface: